You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Mike Thomsen <mi...@gmail.com> on 2019/02/20 13:01:56 UTC
Proposal for ElasticSearch support
I would like to mark the v5 Elastic bundle as deprecated in 1.10. Per
Elastic's official guidelines, the transport API--which it uses--is
deprecated in Elastic 7 and to be removed from at least public
accessibility in Elastic 8.
https://www.elastic.co/guide/en/elasticsearch/client/java-api/master/transport-client.html
So I'd like to nudge any users using it to be prepare to align themselves
more closely with Elastic's roadmap.
ElasticSearch 5.X users can continue to use either the plain vanilla
Elastic bundle or start using the new REST API one going forward. The REST
APIs don't appear to be changing except in adding new features, so I think
we can deprecate and migrate users safely.
(Also, it would free up about 30MB-33MB from our binary distro).
Thoughts?
Mikex
Re: Proposal for ElasticSearch support
Posted by Otto Fowler <ot...@gmail.com>.
I think that there should be specific documentation guidance around this,
like “Picking the right Elasticsearch Processors” to avoid issues.
On February 20, 2019 at 08:02:18, Mike Thomsen (mikerthomsen@gmail.com)
wrote:
I would like to mark the v5 Elastic bundle as deprecated in 1.10. Per
Elastic's official guidelines, the transport API--which it uses--is
deprecated in Elastic 7 and to be removed from at least public
accessibility in Elastic 8.
https://www.elastic.co/guide/en/elasticsearch/client/java-api/master/transport-client.html
So I'd like to nudge any users using it to be prepare to align themselves
more closely with Elastic's roadmap.
ElasticSearch 5.X users can continue to use either the plain vanilla
Elastic bundle or start using the new REST API one going forward. The REST
APIs don't appear to be changing except in adding new features, so I think
we can deprecate and migrate users safely.
(Also, it would free up about 30MB-33MB from our binary distro).
Thoughts?
Mikex
Re: Proposal for ElasticSearch support
Posted by Mike Thomsen <mi...@gmail.com>.
Normally, I would agree with that, but ES is very popular so I wanted to
give users a chance to shout "hey don't do that because [reason]."
On Wed, Feb 20, 2019 at 10:27 AM Joe Witt <jo...@gmail.com> wrote:
> ...probably for dev thread but I am starting to think we should just start
> removing certain nars from the convenience build/assembly and documenting
> it in the migration guide for users that need those. We can then show them
> how to use the hot loading/etc..
>
> Thanks
>
> On Wed, Feb 20, 2019 at 10:04 AM Matt Burgess <ma...@apache.org>
> wrote:
>
>> +1 for deprecating both the v2 and v5 (those using a transport client)
>> components in 1.10, to be removed later. What do you think about
>> refactoring the top-level ES bundle into 3 (v2, v5, REST) and creating
>> profiles (deactivated by default) for the v2 and v5 bundles? I guess
>> that could wait until actual "removal", and we can just tag the
>> components @Deprecated or whatever for 1.10.
>>
>> Also at that point if we had a public Extension Registry we could put
>> those NARs up for folks to download. Note that the NARs are already
>> published anyway (even if not included in the distro), [1] is an
>> example of a NAR that is not included in the distro but is available
>> for download. Just saying an ExtensionRegistry would make that easier
>> :)
>>
>> Regards,
>> Matt
>>
>> [1]
>> https://repository.apache.org/content/repositories/releases/org/apache/nifi/nifi-hive3-nar/1.9.0/
>>
>> On Wed, Feb 20, 2019 at 8:02 AM Mike Thomsen <mi...@gmail.com>
>> wrote:
>> >
>> > I would like to mark the v5 Elastic bundle as deprecated in 1.10. Per
>> Elastic's official guidelines, the transport API--which it uses--is
>> deprecated in Elastic 7 and to be removed from at least public
>> accessibility in Elastic 8.
>> >
>> >
>> https://www.elastic.co/guide/en/elasticsearch/client/java-api/master/transport-client.html
>> >
>> > So I'd like to nudge any users using it to be prepare to align
>> themselves more closely with Elastic's roadmap.
>> >
>> > ElasticSearch 5.X users can continue to use either the plain vanilla
>> Elastic bundle or start using the new REST API one going forward. The REST
>> APIs don't appear to be changing except in adding new features, so I think
>> we can deprecate and migrate users safely.
>> >
>> > (Also, it would free up about 30MB-33MB from our binary distro).
>> >
>> > Thoughts?
>> >
>> > Mikex
>>
>
Re: Proposal for ElasticSearch support
Posted by Otto Fowler <ot...@gmail.com>.
Maybe this would be a nice first use case for that strategy, we can wrap it
up from top to bottom.
On February 20, 2019 at 10:27:40, Joe Witt (joe.witt@gmail.com) wrote:
...probably for dev thread but I am starting to think we should just start
removing certain nars from the convenience build/assembly and documenting
it in the migration guide for users that need those. We can then show them
how to use the hot loading/etc..
Thanks
On Wed, Feb 20, 2019 at 10:04 AM Matt Burgess <ma...@apache.org> wrote:
> +1 for deprecating both the v2 and v5 (those using a transport client)
> components in 1.10, to be removed later. What do you think about
> refactoring the top-level ES bundle into 3 (v2, v5, REST) and creating
> profiles (deactivated by default) for the v2 and v5 bundles? I guess
> that could wait until actual "removal", and we can just tag the
> components @Deprecated or whatever for 1.10.
>
> Also at that point if we had a public Extension Registry we could put
> those NARs up for folks to download. Note that the NARs are already
> published anyway (even if not included in the distro), [1] is an
> example of a NAR that is not included in the distro but is available
> for download. Just saying an ExtensionRegistry would make that easier
> :)
>
> Regards,
> Matt
>
> [1]
> https://repository.apache.org/content/repositories/releases/org/apache/nifi/nifi-hive3-nar/1.9.0/
>
> On Wed, Feb 20, 2019 at 8:02 AM Mike Thomsen <mi...@gmail.com>
> wrote:
> >
> > I would like to mark the v5 Elastic bundle as deprecated in 1.10. Per
> Elastic's official guidelines, the transport API--which it uses--is
> deprecated in Elastic 7 and to be removed from at least public
> accessibility in Elastic 8.
> >
> >
> https://www.elastic.co/guide/en/elasticsearch/client/java-api/master/transport-client.html
> >
> > So I'd like to nudge any users using it to be prepare to align
> themselves more closely with Elastic's roadmap.
> >
> > ElasticSearch 5.X users can continue to use either the plain vanilla
> Elastic bundle or start using the new REST API one going forward. The REST
> APIs don't appear to be changing except in adding new features, so I think
> we can deprecate and migrate users safely.
> >
> > (Also, it would free up about 30MB-33MB from our binary distro).
> >
> > Thoughts?
> >
> > Mikex
>
Re: Proposal for ElasticSearch support
Posted by Joe Witt <jo...@gmail.com>.
...probably for dev thread but I am starting to think we should just start
removing certain nars from the convenience build/assembly and documenting
it in the migration guide for users that need those. We can then show them
how to use the hot loading/etc..
Thanks
On Wed, Feb 20, 2019 at 10:04 AM Matt Burgess <ma...@apache.org> wrote:
> +1 for deprecating both the v2 and v5 (those using a transport client)
> components in 1.10, to be removed later. What do you think about
> refactoring the top-level ES bundle into 3 (v2, v5, REST) and creating
> profiles (deactivated by default) for the v2 and v5 bundles? I guess
> that could wait until actual "removal", and we can just tag the
> components @Deprecated or whatever for 1.10.
>
> Also at that point if we had a public Extension Registry we could put
> those NARs up for folks to download. Note that the NARs are already
> published anyway (even if not included in the distro), [1] is an
> example of a NAR that is not included in the distro but is available
> for download. Just saying an ExtensionRegistry would make that easier
> :)
>
> Regards,
> Matt
>
> [1]
> https://repository.apache.org/content/repositories/releases/org/apache/nifi/nifi-hive3-nar/1.9.0/
>
> On Wed, Feb 20, 2019 at 8:02 AM Mike Thomsen <mi...@gmail.com>
> wrote:
> >
> > I would like to mark the v5 Elastic bundle as deprecated in 1.10. Per
> Elastic's official guidelines, the transport API--which it uses--is
> deprecated in Elastic 7 and to be removed from at least public
> accessibility in Elastic 8.
> >
> >
> https://www.elastic.co/guide/en/elasticsearch/client/java-api/master/transport-client.html
> >
> > So I'd like to nudge any users using it to be prepare to align
> themselves more closely with Elastic's roadmap.
> >
> > ElasticSearch 5.X users can continue to use either the plain vanilla
> Elastic bundle or start using the new REST API one going forward. The REST
> APIs don't appear to be changing except in adding new features, so I think
> we can deprecate and migrate users safely.
> >
> > (Also, it would free up about 30MB-33MB from our binary distro).
> >
> > Thoughts?
> >
> > Mikex
>
Re: Proposal for ElasticSearch support
Posted by Mike Thomsen <mi...@gmail.com>.
Elasticsearch 7.0 is out, and I updated the REST API bundle to have
preliminary support for it. Also removed the high level rest client in
favor of using only the low level one per some documentation provided in
the PR. So going forward that bundle should be a lot more future-proof than
the current bundles.
https://github.com/apache/nifi/pull/2861
The PR should be ready to merge at this point.
On Wed, Feb 20, 2019 at 12:09 PM Mike Thomsen <mi...@gmail.com>
wrote:
> Matt,
>
> I think we could proceed like this:
>
> 1. Deprecate v2 and v5 in 1.10
> 2. Remove v5 from the assembly in 1.11.
> 3. Announce that v2 will be removed from the assembly in 1.13 or
> 1.14--whichever coincides with the end of Q4/early Q1 2020.
>
> That gives us a whole year to work on the REST API bundle as the single
> official way forward that not only uses controller services, but gives
> folks the ability to swap in new Elastic capabilities as the underlying
> APIs change.
>
> On Wed, Feb 20, 2019 at 10:04 AM Matt Burgess <ma...@apache.org>
> wrote:
>
>> +1 for deprecating both the v2 and v5 (those using a transport client)
>> components in 1.10, to be removed later. What do you think about
>> refactoring the top-level ES bundle into 3 (v2, v5, REST) and creating
>> profiles (deactivated by default) for the v2 and v5 bundles? I guess
>> that could wait until actual "removal", and we can just tag the
>> components @Deprecated or whatever for 1.10.
>>
>> Also at that point if we had a public Extension Registry we could put
>> those NARs up for folks to download. Note that the NARs are already
>> published anyway (even if not included in the distro), [1] is an
>> example of a NAR that is not included in the distro but is available
>> for download. Just saying an ExtensionRegistry would make that easier
>> :)
>>
>> Regards,
>> Matt
>>
>> [1]
>> https://repository.apache.org/content/repositories/releases/org/apache/nifi/nifi-hive3-nar/1.9.0/
>>
>> On Wed, Feb 20, 2019 at 8:02 AM Mike Thomsen <mi...@gmail.com>
>> wrote:
>> >
>> > I would like to mark the v5 Elastic bundle as deprecated in 1.10. Per
>> Elastic's official guidelines, the transport API--which it uses--is
>> deprecated in Elastic 7 and to be removed from at least public
>> accessibility in Elastic 8.
>> >
>> >
>> https://www.elastic.co/guide/en/elasticsearch/client/java-api/master/transport-client.html
>> >
>> > So I'd like to nudge any users using it to be prepare to align
>> themselves more closely with Elastic's roadmap.
>> >
>> > ElasticSearch 5.X users can continue to use either the plain vanilla
>> Elastic bundle or start using the new REST API one going forward. The REST
>> APIs don't appear to be changing except in adding new features, so I think
>> we can deprecate and migrate users safely.
>> >
>> > (Also, it would free up about 30MB-33MB from our binary distro).
>> >
>> > Thoughts?
>> >
>> > Mikex
>>
>
Re: Proposal for ElasticSearch support
Posted by Mike Thomsen <mi...@gmail.com>.
Matt,
I think we could proceed like this:
1. Deprecate v2 and v5 in 1.10
2. Remove v5 from the assembly in 1.11.
3. Announce that v2 will be removed from the assembly in 1.13 or
1.14--whichever coincides with the end of Q4/early Q1 2020.
That gives us a whole year to work on the REST API bundle as the single
official way forward that not only uses controller services, but gives
folks the ability to swap in new Elastic capabilities as the underlying
APIs change.
On Wed, Feb 20, 2019 at 10:04 AM Matt Burgess <ma...@apache.org> wrote:
> +1 for deprecating both the v2 and v5 (those using a transport client)
> components in 1.10, to be removed later. What do you think about
> refactoring the top-level ES bundle into 3 (v2, v5, REST) and creating
> profiles (deactivated by default) for the v2 and v5 bundles? I guess
> that could wait until actual "removal", and we can just tag the
> components @Deprecated or whatever for 1.10.
>
> Also at that point if we had a public Extension Registry we could put
> those NARs up for folks to download. Note that the NARs are already
> published anyway (even if not included in the distro), [1] is an
> example of a NAR that is not included in the distro but is available
> for download. Just saying an ExtensionRegistry would make that easier
> :)
>
> Regards,
> Matt
>
> [1]
> https://repository.apache.org/content/repositories/releases/org/apache/nifi/nifi-hive3-nar/1.9.0/
>
> On Wed, Feb 20, 2019 at 8:02 AM Mike Thomsen <mi...@gmail.com>
> wrote:
> >
> > I would like to mark the v5 Elastic bundle as deprecated in 1.10. Per
> Elastic's official guidelines, the transport API--which it uses--is
> deprecated in Elastic 7 and to be removed from at least public
> accessibility in Elastic 8.
> >
> >
> https://www.elastic.co/guide/en/elasticsearch/client/java-api/master/transport-client.html
> >
> > So I'd like to nudge any users using it to be prepare to align
> themselves more closely with Elastic's roadmap.
> >
> > ElasticSearch 5.X users can continue to use either the plain vanilla
> Elastic bundle or start using the new REST API one going forward. The REST
> APIs don't appear to be changing except in adding new features, so I think
> we can deprecate and migrate users safely.
> >
> > (Also, it would free up about 30MB-33MB from our binary distro).
> >
> > Thoughts?
> >
> > Mikex
>
Re: Proposal for ElasticSearch support
Posted by Matt Burgess <ma...@apache.org>.
+1 for deprecating both the v2 and v5 (those using a transport client)
components in 1.10, to be removed later. What do you think about
refactoring the top-level ES bundle into 3 (v2, v5, REST) and creating
profiles (deactivated by default) for the v2 and v5 bundles? I guess
that could wait until actual "removal", and we can just tag the
components @Deprecated or whatever for 1.10.
Also at that point if we had a public Extension Registry we could put
those NARs up for folks to download. Note that the NARs are already
published anyway (even if not included in the distro), [1] is an
example of a NAR that is not included in the distro but is available
for download. Just saying an ExtensionRegistry would make that easier
:)
Regards,
Matt
[1] https://repository.apache.org/content/repositories/releases/org/apache/nifi/nifi-hive3-nar/1.9.0/
On Wed, Feb 20, 2019 at 8:02 AM Mike Thomsen <mi...@gmail.com> wrote:
>
> I would like to mark the v5 Elastic bundle as deprecated in 1.10. Per Elastic's official guidelines, the transport API--which it uses--is deprecated in Elastic 7 and to be removed from at least public accessibility in Elastic 8.
>
> https://www.elastic.co/guide/en/elasticsearch/client/java-api/master/transport-client.html
>
> So I'd like to nudge any users using it to be prepare to align themselves more closely with Elastic's roadmap.
>
> ElasticSearch 5.X users can continue to use either the plain vanilla Elastic bundle or start using the new REST API one going forward. The REST APIs don't appear to be changing except in adding new features, so I think we can deprecate and migrate users safely.
>
> (Also, it would free up about 30MB-33MB from our binary distro).
>
> Thoughts?
>
> Mikex