You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by David Handermann <ex...@apache.org> on 2021/07/23 14:13:10 UTC
[DISCUSS] NiFi 2.0 Release Goals
Team,
With all of the excellent work that many have contributed to NiFi over the
years, the code base has also accumulated some amount of technical debt. A
handful of components have been marked as deprecated, and some components
remain in the code base to support integration with old versions of various
products. Following the principles of semantic versioning, introducing a
major release would provide the opportunity to remove these deprecated and
unsupported components.
Rather than focusing the next major release on new features, what do you
think about focusing on technical debt removal? This approach would not
make for the most interesting release, but it provides the opportunity to
clean up elements that involve breaking changes.
Focusing on technical debt, at least three primary goals come to mind for
the next major release:
1. Removal of deprecated and unmaintained components
2. Require Java 11 as the minimum supported version
3. Transition internal date and time handling to JSR 310 java.time
components
*Removing Deprecated Components*
Removing support for older and deprecated components provides a great
opportunity to improve the overall security posture when it comes to
maintaining dependencies. The OWASP dependency plugin report currently
generates 50 MB of HTML for questionable dependencies, many of which are
related to old versions of various libraries.
As a starting point, here are a handful of components and extension modules
that could be targeted for removal in a major version:
- PostHTTP and GetHTTP
- ListenLumberjack and the entire nifi-lumberjack-bundle
- ListenBeats and the entire nifi-beats-bundle
- Elasticsearch 5 components
- Hive 1 and 2 components
*Requiring Java 11*
Java 8 is now over seven years old, and NiFi has supported general
compatibility with Java 11 for several years. NiFi 1.14.0 incorporated
internal improvements specifically related to TLS 1.3, which allowed
closing out the long-running Java 11 compatibility epic NIFI-5174. Making
Java 11 the minimum required version provides the opportunity to address
any lingering edge cases and put NiFi in a better position to support
current Java versions.
*JSR 310 for Date and Time Handling*
Without making the scope too broad, transitioning internal date and time
handling to use DateTimeFormatter instead of SimpleDateFormat would provide
a number of advantages. The Java Time components provide much better
clarity when it comes to handling localized date and time representations,
and also avoid the inherent confusion of java.sql.Date extending
java.util.Date. Many internal components, specifically Record-oriented
processors and services, rely on date parsing, leading to confusion and
various workarounds. The pattern formats of SimpleDateFormat and
DateTimeFormatter are very similar, but there are a few subtle differences.
Making this transition would provide a much better foundation going forward.
*Conclusion*
Thanks for giving this proposal some consideration. Many of you have been
developing NiFi for years and I look forward to your feedback. I would be
glad to put together a more formalized recommendation on Confluence and
write up Jira epics if this general approach sounds agreeable to the
community.
Regards,
David Handermann
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Jon Logan <jm...@buffalo.edu>.
I would be extremely careful about moving to Java 11 -- especially if NiFi
1.x is not actively maintained. I am sure it's not news to anyone, but a
lot (most?) of people are still on Java 8, for better or worse, and I do
not think moving without a strong, compelling reason is advisable. It's not
as simple to tell users to just update Java -- there is a lot involved with
that.
This is from last year, but probably still proves the point:
https://www.jetbrains.com/lp/devecosystem-2020/java/
On Fri, Jul 23, 2021 at 10:32 AM Joe Witt <jo...@gmail.com> wrote:
> David,
>
> I think this is a highly reasonable approach and such a focus will
> greatly help make a 2.0 release far more approachable to knock out.
> Not only that but tech debt reduction would help make work towards
> major features we'd think about in a 'major release' sense more
> approachable.
>
> We should remove all deprecated things (as well as verify we have the
> right list). We should remove/consider removal of deprecated concepts
> like templates. We should consider whether we can resolve the various
> ways we've handled what are now parameters down to one clean approach.
> We should remove options in the nifi.properties which turn out to
> never be used quite right (if there are). There is quite a bit we can
> do purely in the name of tech debt reduction.
>
> Lots to consider here but I think this is the right discussion.
>
> Than ks
>
> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bb...@gmail.com> wrote:
> >
> > I'm a +1 for this... Not sure if this falls under "Removing Deprecated
> > Components", but I think we should also look at anything that has been
> > marked as deprecated throughout the code base as a candidate for
> > removal. There are quite a few classes, methods, properties, etc that
> > have been waiting for a chance to be removed.
> >
> > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > <ex...@apache.org> wrote:
> > >
> > > Team,
> > >
> > > With all of the excellent work that many have contributed to NiFi over
> the
> > > years, the code base has also accumulated some amount of technical
> debt. A
> > > handful of components have been marked as deprecated, and some
> components
> > > remain in the code base to support integration with old versions of
> various
> > > products. Following the principles of semantic versioning, introducing
> a
> > > major release would provide the opportunity to remove these deprecated
> and
> > > unsupported components.
> > >
> > > Rather than focusing the next major release on new features, what do
> you
> > > think about focusing on technical debt removal? This approach would not
> > > make for the most interesting release, but it provides the opportunity
> to
> > > clean up elements that involve breaking changes.
> > >
> > > Focusing on technical debt, at least three primary goals come to mind
> for
> > > the next major release:
> > >
> > > 1. Removal of deprecated and unmaintained components
> > > 2. Require Java 11 as the minimum supported version
> > > 3. Transition internal date and time handling to JSR 310 java.time
> > > components
> > >
> > > *Removing Deprecated Components*
> > >
> > > Removing support for older and deprecated components provides a great
> > > opportunity to improve the overall security posture when it comes to
> > > maintaining dependencies. The OWASP dependency plugin report currently
> > > generates 50 MB of HTML for questionable dependencies, many of which
> are
> > > related to old versions of various libraries.
> > >
> > > As a starting point, here are a handful of components and extension
> modules
> > > that could be targeted for removal in a major version:
> > >
> > > - PostHTTP and GetHTTP
> > > - ListenLumberjack and the entire nifi-lumberjack-bundle
> > > - ListenBeats and the entire nifi-beats-bundle
> > > - Elasticsearch 5 components
> > > - Hive 1 and 2 components
> > >
> > > *Requiring Java 11*
> > >
> > > Java 8 is now over seven years old, and NiFi has supported general
> > > compatibility with Java 11 for several years. NiFi 1.14.0 incorporated
> > > internal improvements specifically related to TLS 1.3, which allowed
> > > closing out the long-running Java 11 compatibility epic NIFI-5174.
> Making
> > > Java 11 the minimum required version provides the opportunity to
> address
> > > any lingering edge cases and put NiFi in a better position to support
> > > current Java versions.
> > >
> > > *JSR 310 for Date and Time Handling*
> > >
> > > Without making the scope too broad, transitioning internal date and
> time
> > > handling to use DateTimeFormatter instead of SimpleDateFormat would
> provide
> > > a number of advantages. The Java Time components provide much better
> > > clarity when it comes to handling localized date and time
> representations,
> > > and also avoid the inherent confusion of java.sql.Date extending
> > > java.util.Date. Many internal components, specifically Record-oriented
> > > processors and services, rely on date parsing, leading to confusion and
> > > various workarounds. The pattern formats of SimpleDateFormat and
> > > DateTimeFormatter are very similar, but there are a few subtle
> differences.
> > > Making this transition would provide a much better foundation going
> forward.
> > >
> > > *Conclusion*
> > >
> > > Thanks for giving this proposal some consideration. Many of you have
> been
> > > developing NiFi for years and I look forward to your feedback. I would
> be
> > > glad to put together a more formalized recommendation on Confluence and
> > > write up Jira epics if this general approach sounds agreeable to the
> > > community.
> > >
> > > Regards,
> > > David Handermann
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Joey Frazee <jo...@icloud.com.INVALID>.
I quite like this idea. It seemed like the first 2.0 release had been being held out for some bigger innovations (nar registry?), but that has also pushed out making nice breaking changes.
What would be the mechanics here? Just starting feeding all the candidates into JIRA? Listing out candidates in a wiki page first?
-joey
> On Jul 23, 2021, at 7:57 AM, Joe Witt <jo...@gmail.com> wrote:
>
> Jon
>
> You're right we have to be careful and you're right there are still
> significant Java 8 users out there. But we also have to be careful
> about security and sustainability of the codebase. If we had talked
> about this last year when that article came out I'd have agreed it is
> too early. Interestingly that link seems to get updated and I tried
> [1] and found more recent data (not sure how recent). Anyway it
> suggests Java 8 is still the top dog but we see good growth on 11. In
> my $dayjob this aligns to what I'm seeing too. Customers didn't seem
> to care about Java 11 until later half last year and now suddenly it
> is all over the place.
>
> I think once we put out a NiFi 2.0 release we'd see rapid decrease in
> work on the 1.x line just being blunt. We did this many years ago
> with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
> so) but it was purely bug fix/security related bits. We would need to
> do something similar. But feature work would almost certainly go to
> the 2.x line. Maybe there are other workable models but my instinct
> suggests this is likely to follow a similar path.
>
> ...anyway I agree it isn't that easy of a call to dump Java 8. We
> need to make the call in both the interests of the user base and the
> contributor base of the community.
>
> [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
>
>
> Thanks
> Joe
>
>> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <jo...@gmail.com> wrote:
>>
>> Russ
>>
>> Yeah the flow registry is a key part of it. But also now you can
>> download the flow definition in JSON (upload i think is there now
>> too). Templates offered a series of challenges such as we store them
>> in the flow definition which has made flows massive in an unintended
>> way which isn't fun for cluster behavior.
>>
>> We have a couple cases where we headed down a particular concept and
>> came up with better approaches later. We need to reconcile these with
>> the benefit of hindsight, and while being careful to be not overly
>> disruptive to existing users, to reduce the codebase/maintenance
>> burden and allow continued evolution of the project.
>>
>> Thanks
>>
>>> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <ru...@windofkeltia.com> wrote:
>>>
>>> Joe,
>>>
>>> I apologize for the off-topic intrusion, but what replaces templates?
>>> The Registry? Templates rocked and we have used them since 0.5.x.
>>>
>>> Russ
>>>
>>> On 7/23/21 8:31 AM, Joe Witt wrote:
>>>> David,
>>>>
>>>> I think this is a highly reasonable approach and such a focus will
>>>> greatly help make a 2.0 release far more approachable to knock out.
>>>> Not only that but tech debt reduction would help make work towards
>>>> major features we'd think about in a 'major release' sense more
>>>> approachable.
>>>>
>>>> We should remove all deprecated things (as well as verify we have the
>>>> right list). We should remove/consider removal of deprecated concepts
>>>> like templates. We should consider whether we can resolve the various
>>>> ways we've handled what are now parameters down to one clean approach.
>>>> We should remove options in the nifi.properties which turn out to
>>>> never be used quite right (if there are). There is quite a bit we can
>>>> do purely in the name of tech debt reduction.
>>>>
>>>> Lots to consider here but I think this is the right discussion.
>>>>
>>>> Than ks
>>>>
>>>> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bb...@gmail.com> wrote:
>>>>> I'm a +1 for this... Not sure if this falls under "Removing Deprecated
>>>>> Components", but I think we should also look at anything that has been
>>>>> marked as deprecated throughout the code base as a candidate for
>>>>> removal. There are quite a few classes, methods, properties, etc that
>>>>> have been waiting for a chance to be removed.
>>>>>
>>>>> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
>>>>> <ex...@apache.org> wrote:
>>>>>> Team,
>>>>>>
>>>>>> With all of the excellent work that many have contributed to NiFi over the
>>>>>> years, the code base has also accumulated some amount of technical debt. A
>>>>>> handful of components have been marked as deprecated, and some components
>>>>>> remain in the code base to support integration with old versions of various
>>>>>> products. Following the principles of semantic versioning, introducing a
>>>>>> major release would provide the opportunity to remove these deprecated and
>>>>>> unsupported components.
>>>>>>
>>>>>> Rather than focusing the next major release on new features, what do you
>>>>>> think about focusing on technical debt removal? This approach would not
>>>>>> make for the most interesting release, but it provides the opportunity to
>>>>>> clean up elements that involve breaking changes.
>>>>>>
>>>>>> Focusing on technical debt, at least three primary goals come to mind for
>>>>>> the next major release:
>>>>>>
>>>>>> 1. Removal of deprecated and unmaintained components
>>>>>> 2. Require Java 11 as the minimum supported version
>>>>>> 3. Transition internal date and time handling to JSR 310 java.time
>>>>>> components
>>>>>>
>>>>>> *Removing Deprecated Components*
>>>>>>
>>>>>> Removing support for older and deprecated components provides a great
>>>>>> opportunity to improve the overall security posture when it comes to
>>>>>> maintaining dependencies. The OWASP dependency plugin report currently
>>>>>> generates 50 MB of HTML for questionable dependencies, many of which are
>>>>>> related to old versions of various libraries.
>>>>>>
>>>>>> As a starting point, here are a handful of components and extension modules
>>>>>> that could be targeted for removal in a major version:
>>>>>>
>>>>>> - PostHTTP and GetHTTP
>>>>>> - ListenLumberjack and the entire nifi-lumberjack-bundle
>>>>>> - ListenBeats and the entire nifi-beats-bundle
>>>>>> - Elasticsearch 5 components
>>>>>> - Hive 1 and 2 components
>>>>>>
>>>>>> *Requiring Java 11*
>>>>>>
>>>>>> Java 8 is now over seven years old, and NiFi has supported general
>>>>>> compatibility with Java 11 for several years. NiFi 1.14.0 incorporated
>>>>>> internal improvements specifically related to TLS 1.3, which allowed
>>>>>> closing out the long-running Java 11 compatibility epic NIFI-5174. Making
>>>>>> Java 11 the minimum required version provides the opportunity to address
>>>>>> any lingering edge cases and put NiFi in a better position to support
>>>>>> current Java versions.
>>>>>>
>>>>>> *JSR 310 for Date and Time Handling*
>>>>>>
>>>>>> Without making the scope too broad, transitioning internal date and time
>>>>>> handling to use DateTimeFormatter instead of SimpleDateFormat would provide
>>>>>> a number of advantages. The Java Time components provide much better
>>>>>> clarity when it comes to handling localized date and time representations,
>>>>>> and also avoid the inherent confusion of java.sql.Date extending
>>>>>> java.util.Date. Many internal components, specifically Record-oriented
>>>>>> processors and services, rely on date parsing, leading to confusion and
>>>>>> various workarounds. The pattern formats of SimpleDateFormat and
>>>>>> DateTimeFormatter are very similar, but there are a few subtle differences.
>>>>>> Making this transition would provide a much better foundation going forward.
>>>>>>
>>>>>> *Conclusion*
>>>>>>
>>>>>> Thanks for giving this proposal some consideration. Many of you have been
>>>>>> developing NiFi for years and I look forward to your feedback. I would be
>>>>>> glad to put together a more formalized recommendation on Confluence and
>>>>>> write up Jira epics if this general approach sounds agreeable to the
>>>>>> community.
>>>>>>
>>>>>> Regards,
>>>>>> David Handermann
>>>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Joe Witt <jo...@gmail.com>.
I'd say we'll have to address the branching strategy but not yet.
Let's get a sense of the scope/specific bits we want to tackle in 2.0
then worry about best way to go about it.
On Fri, Jul 23, 2021 at 9:39 AM David Handermann
<ex...@gmail.com> wrote:
>
> Thanks to everyone who has provided feedback thus far, it is good to see
> general interest in moving forward with a release focusing on technical
> debt removal.
>
> It seems like it would be helpful to start gathering removal candidates on
> a Confluence wiki page, and then turning those into Jira issues. I am open
> to suggestions here, but it seems like creating several epics related to
> high level goals would be a good way to organize the work. I can start with
> creating a Confluence wiki page to capture some of these initial details.
>
> A release version 2.0.0 already exists in Jira, so that could be used for
> tagging planned issues.
>
> From a branching and maintenance perspective, perhaps PMC members could
> describe how this should work? Should the main branch become the branch
> planned for 2.0.0 work, and does that require a vote for approval to
> proceed? Or should a new 2.0 branch be created and new commits applied
> selectively?
>
> Regards,
> David Handermann
>
> On Fri, Jul 23, 2021 at 11:29 AM Matt Burgess <ma...@apache.org> wrote:
>
> > Along with the itemized list for ancient components we should look at
> > updating versions of drivers, SDKs, etc. for external systems such as
> > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0
> > is probably the right time to get things up to date to make them more
> > useful to more people.
> >
> > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <th...@gmail.com> wrote:
> > >
> > > I'm a +1 for removing pretty much all of this stuff. There are security
> > > implications to keeping old dependencies around, so the more old code we
> > > can remove the better. I agree that eventually we need to move to
> > > supporting only Java 11+, and as our next release will probably be about
> > 4
> > > - 6 months from now that doesn't seem too soon. We could potentially
> > break
> > > this in two and remove the deprecated processors and leave 1.x on Java 8,
> > > and finally start on 2.x which would support only Java 11. I'm unsure of
> > > what implications changing the date and time handling would have - for
> > > running systems that use long term historical logs, unexpected impacts to
> > > time logging could be a problem.
> > >
> > > As Joe says I think feature work will have to be dedicated to 2.x and we
> > > could support 1.x for security fixes for some period of time. 2.x seems
> > > like a gargantuan task but it's probably time to get started. Not sure
> > how
> > > we handle all open PRs and the transition between 1.x and 2.x.
> > >
> > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <jo...@gmail.com> wrote:
> > >
> > > > Jon
> > > >
> > > > You're right we have to be careful and you're right there are still
> > > > significant Java 8 users out there. But we also have to be careful
> > > > about security and sustainability of the codebase. If we had talked
> > > > about this last year when that article came out I'd have agreed it is
> > > > too early. Interestingly that link seems to get updated and I tried
> > > > [1] and found more recent data (not sure how recent). Anyway it
> > > > suggests Java 8 is still the top dog but we see good growth on 11. In
> > > > my $dayjob this aligns to what I'm seeing too. Customers didn't seem
> > > > to care about Java 11 until later half last year and now suddenly it
> > > > is all over the place.
> > > >
> > > > I think once we put out a NiFi 2.0 release we'd see rapid decrease in
> > > > work on the 1.x line just being blunt. We did this many years ago
> > > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
> > > > so) but it was purely bug fix/security related bits. We would need to
> > > > do something similar. But feature work would almost certainly go to
> > > > the 2.x line. Maybe there are other workable models but my instinct
> > > > suggests this is likely to follow a similar path.
> > > >
> > > > ...anyway I agree it isn't that easy of a call to dump Java 8. We
> > > > need to make the call in both the interests of the user base and the
> > > > contributor base of the community.
> > > >
> > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > >
> > > >
> > > > Thanks
> > > > Joe
> > > >
> > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <jo...@gmail.com> wrote:
> > > > >
> > > > > Russ
> > > > >
> > > > > Yeah the flow registry is a key part of it. But also now you can
> > > > > download the flow definition in JSON (upload i think is there now
> > > > > too). Templates offered a series of challenges such as we store them
> > > > > in the flow definition which has made flows massive in an unintended
> > > > > way which isn't fun for cluster behavior.
> > > > >
> > > > > We have a couple cases where we headed down a particular concept and
> > > > > came up with better approaches later. We need to reconcile these
> > with
> > > > > the benefit of hindsight, and while being careful to be not overly
> > > > > disruptive to existing users, to reduce the codebase/maintenance
> > > > > burden and allow continued evolution of the project.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > russ@windofkeltia.com>
> > > > wrote:
> > > > > >
> > > > > > Joe,
> > > > > >
> > > > > > I apologize for the off-topic intrusion, but what replaces
> > templates?
> > > > > > The Registry? Templates rocked and we have used them since 0.5.x.
> > > > > >
> > > > > > Russ
> > > > > >
> > > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > > > David,
> > > > > > >
> > > > > > > I think this is a highly reasonable approach and such a focus
> > will
> > > > > > > greatly help make a 2.0 release far more approachable to knock
> > out.
> > > > > > > Not only that but tech debt reduction would help make work
> > towards
> > > > > > > major features we'd think about in a 'major release' sense more
> > > > > > > approachable.
> > > > > > >
> > > > > > > We should remove all deprecated things (as well as verify we
> > have the
> > > > > > > right list). We should remove/consider removal of deprecated
> > > > concepts
> > > > > > > like templates. We should consider whether we can resolve the
> > > > various
> > > > > > > ways we've handled what are now parameters down to one clean
> > > > approach.
> > > > > > > We should remove options in the nifi.properties which turn out to
> > > > > > > never be used quite right (if there are). There is quite a bit
> > we
> > > > can
> > > > > > > do purely in the name of tech debt reduction.
> > > > > > >
> > > > > > > Lots to consider here but I think this is the right discussion.
> > > > > > >
> > > > > > > Than ks
> > > > > > >
> > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bb...@gmail.com>
> > > > wrote:
> > > > > > >> I'm a +1 for this... Not sure if this falls under "Removing
> > > > Deprecated
> > > > > > >> Components", but I think we should also look at anything that
> > has
> > > > been
> > > > > > >> marked as deprecated throughout the code base as a candidate for
> > > > > > >> removal. There are quite a few classes, methods, properties, etc
> > > > that
> > > > > > >> have been waiting for a chance to be removed.
> > > > > > >>
> > > > > > >> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > > > >> <ex...@apache.org> wrote:
> > > > > > >>> Team,
> > > > > > >>>
> > > > > > >>> With all of the excellent work that many have contributed to
> > NiFi
> > > > over the
> > > > > > >>> years, the code base has also accumulated some amount of
> > technical
> > > > debt. A
> > > > > > >>> handful of components have been marked as deprecated, and some
> > > > components
> > > > > > >>> remain in the code base to support integration with old
> > versions
> > > > of various
> > > > > > >>> products. Following the principles of semantic versioning,
> > > > introducing a
> > > > > > >>> major release would provide the opportunity to remove these
> > > > deprecated and
> > > > > > >>> unsupported components.
> > > > > > >>>
> > > > > > >>> Rather than focusing the next major release on new features,
> > what
> > > > do you
> > > > > > >>> think about focusing on technical debt removal? This approach
> > > > would not
> > > > > > >>> make for the most interesting release, but it provides the
> > > > opportunity to
> > > > > > >>> clean up elements that involve breaking changes.
> > > > > > >>>
> > > > > > >>> Focusing on technical debt, at least three primary goals come
> > to
> > > > mind for
> > > > > > >>> the next major release:
> > > > > > >>>
> > > > > > >>> 1. Removal of deprecated and unmaintained components
> > > > > > >>> 2. Require Java 11 as the minimum supported version
> > > > > > >>> 3. Transition internal date and time handling to JSR 310
> > java.time
> > > > > > >>> components
> > > > > > >>>
> > > > > > >>> *Removing Deprecated Components*
> > > > > > >>>
> > > > > > >>> Removing support for older and deprecated components provides a
> > > > great
> > > > > > >>> opportunity to improve the overall security posture when it
> > comes
> > > > to
> > > > > > >>> maintaining dependencies. The OWASP dependency plugin report
> > > > currently
> > > > > > >>> generates 50 MB of HTML for questionable dependencies, many of
> > > > which are
> > > > > > >>> related to old versions of various libraries.
> > > > > > >>>
> > > > > > >>> As a starting point, here are a handful of components and
> > > > extension modules
> > > > > > >>> that could be targeted for removal in a major version:
> > > > > > >>>
> > > > > > >>> - PostHTTP and GetHTTP
> > > > > > >>> - ListenLumberjack and the entire nifi-lumberjack-bundle
> > > > > > >>> - ListenBeats and the entire nifi-beats-bundle
> > > > > > >>> - Elasticsearch 5 components
> > > > > > >>> - Hive 1 and 2 components
> > > > > > >>>
> > > > > > >>> *Requiring Java 11*
> > > > > > >>>
> > > > > > >>> Java 8 is now over seven years old, and NiFi has supported
> > general
> > > > > > >>> compatibility with Java 11 for several years. NiFi 1.14.0
> > > > incorporated
> > > > > > >>> internal improvements specifically related to TLS 1.3, which
> > > > allowed
> > > > > > >>> closing out the long-running Java 11 compatibility epic
> > NIFI-5174.
> > > > Making
> > > > > > >>> Java 11 the minimum required version provides the opportunity
> > to
> > > > address
> > > > > > >>> any lingering edge cases and put NiFi in a better position to
> > > > support
> > > > > > >>> current Java versions.
> > > > > > >>>
> > > > > > >>> *JSR 310 for Date and Time Handling*
> > > > > > >>>
> > > > > > >>> Without making the scope too broad, transitioning internal date
> > > > and time
> > > > > > >>> handling to use DateTimeFormatter instead of SimpleDateFormat
> > > > would provide
> > > > > > >>> a number of advantages. The Java Time components provide much
> > > > better
> > > > > > >>> clarity when it comes to handling localized date and time
> > > > representations,
> > > > > > >>> and also avoid the inherent confusion of java.sql.Date
> > extending
> > > > > > >>> java.util.Date. Many internal components, specifically
> > > > Record-oriented
> > > > > > >>> processors and services, rely on date parsing, leading to
> > > > confusion and
> > > > > > >>> various workarounds. The pattern formats of SimpleDateFormat
> > and
> > > > > > >>> DateTimeFormatter are very similar, but there are a few subtle
> > > > differences.
> > > > > > >>> Making this transition would provide a much better foundation
> > > > going forward.
> > > > > > >>>
> > > > > > >>> *Conclusion*
> > > > > > >>>
> > > > > > >>> Thanks for giving this proposal some consideration. Many of you
> > > > have been
> > > > > > >>> developing NiFi for years and I look forward to your feedback.
> > I
> > > > would be
> > > > > > >>> glad to put together a more formalized recommendation on
> > > > Confluence and
> > > > > > >>> write up Jira epics if this general approach sounds agreeable
> > to
> > > > the
> > > > > > >>> community.
> > > > > > >>>
> > > > > > >>> Regards,
> > > > > > >>> David Handermann
> > > > > >
> > > >
> >
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by David Handermann <ex...@gmail.com>.
Thanks to everyone who has provided feedback thus far, it is good to see
general interest in moving forward with a release focusing on technical
debt removal.
It seems like it would be helpful to start gathering removal candidates on
a Confluence wiki page, and then turning those into Jira issues. I am open
to suggestions here, but it seems like creating several epics related to
high level goals would be a good way to organize the work. I can start with
creating a Confluence wiki page to capture some of these initial details.
A release version 2.0.0 already exists in Jira, so that could be used for
tagging planned issues.
From a branching and maintenance perspective, perhaps PMC members could
describe how this should work? Should the main branch become the branch
planned for 2.0.0 work, and does that require a vote for approval to
proceed? Or should a new 2.0 branch be created and new commits applied
selectively?
Regards,
David Handermann
On Fri, Jul 23, 2021 at 11:29 AM Matt Burgess <ma...@apache.org> wrote:
> Along with the itemized list for ancient components we should look at
> updating versions of drivers, SDKs, etc. for external systems such as
> Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0
> is probably the right time to get things up to date to make them more
> useful to more people.
>
> On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <th...@gmail.com> wrote:
> >
> > I'm a +1 for removing pretty much all of this stuff. There are security
> > implications to keeping old dependencies around, so the more old code we
> > can remove the better. I agree that eventually we need to move to
> > supporting only Java 11+, and as our next release will probably be about
> 4
> > - 6 months from now that doesn't seem too soon. We could potentially
> break
> > this in two and remove the deprecated processors and leave 1.x on Java 8,
> > and finally start on 2.x which would support only Java 11. I'm unsure of
> > what implications changing the date and time handling would have - for
> > running systems that use long term historical logs, unexpected impacts to
> > time logging could be a problem.
> >
> > As Joe says I think feature work will have to be dedicated to 2.x and we
> > could support 1.x for security fixes for some period of time. 2.x seems
> > like a gargantuan task but it's probably time to get started. Not sure
> how
> > we handle all open PRs and the transition between 1.x and 2.x.
> >
> > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <jo...@gmail.com> wrote:
> >
> > > Jon
> > >
> > > You're right we have to be careful and you're right there are still
> > > significant Java 8 users out there. But we also have to be careful
> > > about security and sustainability of the codebase. If we had talked
> > > about this last year when that article came out I'd have agreed it is
> > > too early. Interestingly that link seems to get updated and I tried
> > > [1] and found more recent data (not sure how recent). Anyway it
> > > suggests Java 8 is still the top dog but we see good growth on 11. In
> > > my $dayjob this aligns to what I'm seeing too. Customers didn't seem
> > > to care about Java 11 until later half last year and now suddenly it
> > > is all over the place.
> > >
> > > I think once we put out a NiFi 2.0 release we'd see rapid decrease in
> > > work on the 1.x line just being blunt. We did this many years ago
> > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
> > > so) but it was purely bug fix/security related bits. We would need to
> > > do something similar. But feature work would almost certainly go to
> > > the 2.x line. Maybe there are other workable models but my instinct
> > > suggests this is likely to follow a similar path.
> > >
> > > ...anyway I agree it isn't that easy of a call to dump Java 8. We
> > > need to make the call in both the interests of the user base and the
> > > contributor base of the community.
> > >
> > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> > >
> > >
> > > Thanks
> > > Joe
> > >
> > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <jo...@gmail.com> wrote:
> > > >
> > > > Russ
> > > >
> > > > Yeah the flow registry is a key part of it. But also now you can
> > > > download the flow definition in JSON (upload i think is there now
> > > > too). Templates offered a series of challenges such as we store them
> > > > in the flow definition which has made flows massive in an unintended
> > > > way which isn't fun for cluster behavior.
> > > >
> > > > We have a couple cases where we headed down a particular concept and
> > > > came up with better approaches later. We need to reconcile these
> with
> > > > the benefit of hindsight, and while being careful to be not overly
> > > > disruptive to existing users, to reduce the codebase/maintenance
> > > > burden and allow continued evolution of the project.
> > > >
> > > > Thanks
> > > >
> > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> russ@windofkeltia.com>
> > > wrote:
> > > > >
> > > > > Joe,
> > > > >
> > > > > I apologize for the off-topic intrusion, but what replaces
> templates?
> > > > > The Registry? Templates rocked and we have used them since 0.5.x.
> > > > >
> > > > > Russ
> > > > >
> > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > > David,
> > > > > >
> > > > > > I think this is a highly reasonable approach and such a focus
> will
> > > > > > greatly help make a 2.0 release far more approachable to knock
> out.
> > > > > > Not only that but tech debt reduction would help make work
> towards
> > > > > > major features we'd think about in a 'major release' sense more
> > > > > > approachable.
> > > > > >
> > > > > > We should remove all deprecated things (as well as verify we
> have the
> > > > > > right list). We should remove/consider removal of deprecated
> > > concepts
> > > > > > like templates. We should consider whether we can resolve the
> > > various
> > > > > > ways we've handled what are now parameters down to one clean
> > > approach.
> > > > > > We should remove options in the nifi.properties which turn out to
> > > > > > never be used quite right (if there are). There is quite a bit
> we
> > > can
> > > > > > do purely in the name of tech debt reduction.
> > > > > >
> > > > > > Lots to consider here but I think this is the right discussion.
> > > > > >
> > > > > > Than ks
> > > > > >
> > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bb...@gmail.com>
> > > wrote:
> > > > > >> I'm a +1 for this... Not sure if this falls under "Removing
> > > Deprecated
> > > > > >> Components", but I think we should also look at anything that
> has
> > > been
> > > > > >> marked as deprecated throughout the code base as a candidate for
> > > > > >> removal. There are quite a few classes, methods, properties, etc
> > > that
> > > > > >> have been waiting for a chance to be removed.
> > > > > >>
> > > > > >> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > > >> <ex...@apache.org> wrote:
> > > > > >>> Team,
> > > > > >>>
> > > > > >>> With all of the excellent work that many have contributed to
> NiFi
> > > over the
> > > > > >>> years, the code base has also accumulated some amount of
> technical
> > > debt. A
> > > > > >>> handful of components have been marked as deprecated, and some
> > > components
> > > > > >>> remain in the code base to support integration with old
> versions
> > > of various
> > > > > >>> products. Following the principles of semantic versioning,
> > > introducing a
> > > > > >>> major release would provide the opportunity to remove these
> > > deprecated and
> > > > > >>> unsupported components.
> > > > > >>>
> > > > > >>> Rather than focusing the next major release on new features,
> what
> > > do you
> > > > > >>> think about focusing on technical debt removal? This approach
> > > would not
> > > > > >>> make for the most interesting release, but it provides the
> > > opportunity to
> > > > > >>> clean up elements that involve breaking changes.
> > > > > >>>
> > > > > >>> Focusing on technical debt, at least three primary goals come
> to
> > > mind for
> > > > > >>> the next major release:
> > > > > >>>
> > > > > >>> 1. Removal of deprecated and unmaintained components
> > > > > >>> 2. Require Java 11 as the minimum supported version
> > > > > >>> 3. Transition internal date and time handling to JSR 310
> java.time
> > > > > >>> components
> > > > > >>>
> > > > > >>> *Removing Deprecated Components*
> > > > > >>>
> > > > > >>> Removing support for older and deprecated components provides a
> > > great
> > > > > >>> opportunity to improve the overall security posture when it
> comes
> > > to
> > > > > >>> maintaining dependencies. The OWASP dependency plugin report
> > > currently
> > > > > >>> generates 50 MB of HTML for questionable dependencies, many of
> > > which are
> > > > > >>> related to old versions of various libraries.
> > > > > >>>
> > > > > >>> As a starting point, here are a handful of components and
> > > extension modules
> > > > > >>> that could be targeted for removal in a major version:
> > > > > >>>
> > > > > >>> - PostHTTP and GetHTTP
> > > > > >>> - ListenLumberjack and the entire nifi-lumberjack-bundle
> > > > > >>> - ListenBeats and the entire nifi-beats-bundle
> > > > > >>> - Elasticsearch 5 components
> > > > > >>> - Hive 1 and 2 components
> > > > > >>>
> > > > > >>> *Requiring Java 11*
> > > > > >>>
> > > > > >>> Java 8 is now over seven years old, and NiFi has supported
> general
> > > > > >>> compatibility with Java 11 for several years. NiFi 1.14.0
> > > incorporated
> > > > > >>> internal improvements specifically related to TLS 1.3, which
> > > allowed
> > > > > >>> closing out the long-running Java 11 compatibility epic
> NIFI-5174.
> > > Making
> > > > > >>> Java 11 the minimum required version provides the opportunity
> to
> > > address
> > > > > >>> any lingering edge cases and put NiFi in a better position to
> > > support
> > > > > >>> current Java versions.
> > > > > >>>
> > > > > >>> *JSR 310 for Date and Time Handling*
> > > > > >>>
> > > > > >>> Without making the scope too broad, transitioning internal date
> > > and time
> > > > > >>> handling to use DateTimeFormatter instead of SimpleDateFormat
> > > would provide
> > > > > >>> a number of advantages. The Java Time components provide much
> > > better
> > > > > >>> clarity when it comes to handling localized date and time
> > > representations,
> > > > > >>> and also avoid the inherent confusion of java.sql.Date
> extending
> > > > > >>> java.util.Date. Many internal components, specifically
> > > Record-oriented
> > > > > >>> processors and services, rely on date parsing, leading to
> > > confusion and
> > > > > >>> various workarounds. The pattern formats of SimpleDateFormat
> and
> > > > > >>> DateTimeFormatter are very similar, but there are a few subtle
> > > differences.
> > > > > >>> Making this transition would provide a much better foundation
> > > going forward.
> > > > > >>>
> > > > > >>> *Conclusion*
> > > > > >>>
> > > > > >>> Thanks for giving this proposal some consideration. Many of you
> > > have been
> > > > > >>> developing NiFi for years and I look forward to your feedback.
> I
> > > would be
> > > > > >>> glad to put together a more formalized recommendation on
> > > Confluence and
> > > > > >>> write up Jira epics if this general approach sounds agreeable
> to
> > > the
> > > > > >>> community.
> > > > > >>>
> > > > > >>> Regards,
> > > > > >>> David Handermann
> > > > >
> > >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by David Handermann <ex...@apache.org>.
Team,
I created the following page on the Apache NiFi wiki to track proposed
goals and particular focus areas. If the page should go under another
section of the wiki, it can be moved.
https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Proposed+Release+Goals
The Primary Goals section is intended to cover high-level categories, and
each subsection includes some initial elements that seem to fit in that
category.
This initial version is not intended to be exhaustive or definitive, but
should keep this conversation moving forward. I can make updates based on
feedback in this thread, and others with access are welcome to make direct
changes. As a list of proposed goals, the purpose is not to exclude
additional ideas, but to promote a focused discussion toward the next major
release. I would be glad to help write up Jira issues when we get to that
point.
For now, more feedback is better in terms of what should be covered under
the general heading of technical debt reduction.
Regards,
David Handermann
On Fri, Jul 23, 2021 at 12:11 PM Russell Bateman <ru...@windofkeltia.com>
wrote:
> Bringing up Elastic also reminds me that the Elastic framework has just
> recently transitioned out of Open Source, so to acknowledge that, maybe
> some effort toward OpenSearch--I say this not understanding exactly how
> this sort of thing is considered in a large-scale, world-class software
> project like Apache NiFi. (I'm not a contributor, just a grateful
> consumer.)
>
> Russ
>
> On 7/23/21 10:28 AM, Matt Burgess wrote:
> > Along with the itemized list for ancient components we should look at
> > updating versions of drivers, SDKs, etc. for external systems such as
> > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0
> > is probably the right time to get things up to date to make them more
> > useful to more people.
> >
> > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <th...@gmail.com>
> wrote:
> >> I'm a +1 for removing pretty much all of this stuff. There are security
> >> implications to keeping old dependencies around, so the more old code we
> >> can remove the better. I agree that eventually we need to move to
> >> supporting only Java 11+, and as our next release will probably be
> about 4
> >> - 6 months from now that doesn't seem too soon. We could potentially
> break
> >> this in two and remove the deprecated processors and leave 1.x on Java
> 8,
> >> and finally start on 2.x which would support only Java 11. I'm unsure of
> >> what implications changing the date and time handling would have - for
> >> running systems that use long term historical logs, unexpected impacts
> to
> >> time logging could be a problem.
> >>
> >> As Joe says I think feature work will have to be dedicated to 2.x and we
> >> could support 1.x for security fixes for some period of time. 2.x seems
> >> like a gargantuan task but it's probably time to get started. Not sure
> how
> >> we handle all open PRs and the transition between 1.x and 2.x.
> >>
> >> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <jo...@gmail.com> wrote:
> >>
> >>> Jon
> >>>
> >>> You're right we have to be careful and you're right there are still
> >>> significant Java 8 users out there. But we also have to be careful
> >>> about security and sustainability of the codebase. If we had talked
> >>> about this last year when that article came out I'd have agreed it is
> >>> too early. Interestingly that link seems to get updated and I tried
> >>> [1] and found more recent data (not sure how recent). Anyway it
> >>> suggests Java 8 is still the top dog but we see good growth on 11. In
> >>> my $dayjob this aligns to what I'm seeing too. Customers didn't seem
> >>> to care about Java 11 until later half last year and now suddenly it
> >>> is all over the place.
> >>>
> >>> I think once we put out a NiFi 2.0 release we'd see rapid decrease in
> >>> work on the 1.x line just being blunt. We did this many years ago
> >>> with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
> >>> so) but it was purely bug fix/security related bits. We would need to
> >>> do something similar. But feature work would almost certainly go to
> >>> the 2.x line. Maybe there are other workable models but my instinct
> >>> suggests this is likely to follow a similar path.
> >>>
> >>> ...anyway I agree it isn't that easy of a call to dump Java 8. We
> >>> need to make the call in both the interests of the user base and the
> >>> contributor base of the community.
> >>>
> >>> [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> >>>
> >>>
> >>> Thanks
> >>> Joe
> >>>
> >>> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <jo...@gmail.com> wrote:
> >>>> Russ
> >>>>
> >>>> Yeah the flow registry is a key part of it. But also now you can
> >>>> download the flow definition in JSON (upload i think is there now
> >>>> too). Templates offered a series of challenges such as we store them
> >>>> in the flow definition which has made flows massive in an unintended
> >>>> way which isn't fun for cluster behavior.
> >>>>
> >>>> We have a couple cases where we headed down a particular concept and
> >>>> came up with better approaches later. We need to reconcile these with
> >>>> the benefit of hindsight, and while being careful to be not overly
> >>>> disruptive to existing users, to reduce the codebase/maintenance
> >>>> burden and allow continued evolution of the project.
> >>>>
> >>>> Thanks
> >>>>
> >>>> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> russ@windofkeltia.com>
> >>> wrote:
> >>>>> Joe,
> >>>>>
> >>>>> I apologize for the off-topic intrusion, but what replaces templates?
> >>>>> The Registry? Templates rocked and we have used them since 0.5.x.
> >>>>>
> >>>>> Russ
> >>>>>
> >>>>> On 7/23/21 8:31 AM, Joe Witt wrote:
> >>>>>> David,
> >>>>>>
> >>>>>> I think this is a highly reasonable approach and such a focus will
> >>>>>> greatly help make a 2.0 release far more approachable to knock out.
> >>>>>> Not only that but tech debt reduction would help make work towards
> >>>>>> major features we'd think about in a 'major release' sense more
> >>>>>> approachable.
> >>>>>>
> >>>>>> We should remove all deprecated things (as well as verify we have
> the
> >>>>>> right list). We should remove/consider removal of deprecated
> >>> concepts
> >>>>>> like templates. We should consider whether we can resolve the
> >>> various
> >>>>>> ways we've handled what are now parameters down to one clean
> >>> approach.
> >>>>>> We should remove options in the nifi.properties which turn out to
> >>>>>> never be used quite right (if there are). There is quite a bit we
> >>> can
> >>>>>> do purely in the name of tech debt reduction.
> >>>>>>
> >>>>>> Lots to consider here but I think this is the right discussion.
> >>>>>>
> >>>>>> Than ks
> >>>>>>
> >>>>>> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bb...@gmail.com>
> >>> wrote:
> >>>>>>> I'm a +1 for this... Not sure if this falls under "Removing
> >>> Deprecated
> >>>>>>> Components", but I think we should also look at anything that has
> >>> been
> >>>>>>> marked as deprecated throughout the code base as a candidate for
> >>>>>>> removal. There are quite a few classes, methods, properties, etc
> >>> that
> >>>>>>> have been waiting for a chance to be removed.
> >>>>>>>
> >>>>>>> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> >>>>>>> <ex...@apache.org> wrote:
> >>>>>>>> Team,
> >>>>>>>>
> >>>>>>>> With all of the excellent work that many have contributed to NiFi
> >>> over the
> >>>>>>>> years, the code base has also accumulated some amount of technical
> >>> debt. A
> >>>>>>>> handful of components have been marked as deprecated, and some
> >>> components
> >>>>>>>> remain in the code base to support integration with old versions
> >>> of various
> >>>>>>>> products. Following the principles of semantic versioning,
> >>> introducing a
> >>>>>>>> major release would provide the opportunity to remove these
> >>> deprecated and
> >>>>>>>> unsupported components.
> >>>>>>>>
> >>>>>>>> Rather than focusing the next major release on new features, what
> >>> do you
> >>>>>>>> think about focusing on technical debt removal? This approach
> >>> would not
> >>>>>>>> make for the most interesting release, but it provides the
> >>> opportunity to
> >>>>>>>> clean up elements that involve breaking changes.
> >>>>>>>>
> >>>>>>>> Focusing on technical debt, at least three primary goals come to
> >>> mind for
> >>>>>>>> the next major release:
> >>>>>>>>
> >>>>>>>> 1. Removal of deprecated and unmaintained components
> >>>>>>>> 2. Require Java 11 as the minimum supported version
> >>>>>>>> 3. Transition internal date and time handling to JSR 310 java.time
> >>>>>>>> components
> >>>>>>>>
> >>>>>>>> *Removing Deprecated Components*
> >>>>>>>>
> >>>>>>>> Removing support for older and deprecated components provides a
> >>> great
> >>>>>>>> opportunity to improve the overall security posture when it comes
> >>> to
> >>>>>>>> maintaining dependencies. The OWASP dependency plugin report
> >>> currently
> >>>>>>>> generates 50 MB of HTML for questionable dependencies, many of
> >>> which are
> >>>>>>>> related to old versions of various libraries.
> >>>>>>>>
> >>>>>>>> As a starting point, here are a handful of components and
> >>> extension modules
> >>>>>>>> that could be targeted for removal in a major version:
> >>>>>>>>
> >>>>>>>> - PostHTTP and GetHTTP
> >>>>>>>> - ListenLumberjack and the entire nifi-lumberjack-bundle
> >>>>>>>> - ListenBeats and the entire nifi-beats-bundle
> >>>>>>>> - Elasticsearch 5 components
> >>>>>>>> - Hive 1 and 2 components
> >>>>>>>>
> >>>>>>>> *Requiring Java 11*
> >>>>>>>>
> >>>>>>>> Java 8 is now over seven years old, and NiFi has supported general
> >>>>>>>> compatibility with Java 11 for several years. NiFi 1.14.0
> >>> incorporated
> >>>>>>>> internal improvements specifically related to TLS 1.3, which
> >>> allowed
> >>>>>>>> closing out the long-running Java 11 compatibility epic NIFI-5174.
> >>> Making
> >>>>>>>> Java 11 the minimum required version provides the opportunity to
> >>> address
> >>>>>>>> any lingering edge cases and put NiFi in a better position to
> >>> support
> >>>>>>>> current Java versions.
> >>>>>>>>
> >>>>>>>> *JSR 310 for Date and Time Handling*
> >>>>>>>>
> >>>>>>>> Without making the scope too broad, transitioning internal date
> >>> and time
> >>>>>>>> handling to use DateTimeFormatter instead of SimpleDateFormat
> >>> would provide
> >>>>>>>> a number of advantages. The Java Time components provide much
> >>> better
> >>>>>>>> clarity when it comes to handling localized date and time
> >>> representations,
> >>>>>>>> and also avoid the inherent confusion of java.sql.Date extending
> >>>>>>>> java.util.Date. Many internal components, specifically
> >>> Record-oriented
> >>>>>>>> processors and services, rely on date parsing, leading to
> >>> confusion and
> >>>>>>>> various workarounds. The pattern formats of SimpleDateFormat and
> >>>>>>>> DateTimeFormatter are very similar, but there are a few subtle
> >>> differences.
> >>>>>>>> Making this transition would provide a much better foundation
> >>> going forward.
> >>>>>>>> *Conclusion*
> >>>>>>>>
> >>>>>>>> Thanks for giving this proposal some consideration. Many of you
> >>> have been
> >>>>>>>> developing NiFi for years and I look forward to your feedback. I
> >>> would be
> >>>>>>>> glad to put together a more formalized recommendation on
> >>> Confluence and
> >>>>>>>> write up Jira epics if this general approach sounds agreeable to
> >>> the
> >>>>>>>> community.
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> David Handermann
>
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Ryan Hendrickson <ry...@gmail.com>.
Thanks for the push Otto. They are created:
[NIFI-9026 <https://issues.apache.org/jira/browse/NIFI-9026>] - Integration
with auto-scaling container environments
[NIFI-9027 <https://issues.apache.org/jira/browse/NIFI-9027>] - Integration
of S3 buckets with Remote Process Groups for outage events and data overruns
[NIFI-9028 <https://issues.apache.org/jira/browse/NIFI-9028>] - Large
Canvas Management
Best,
Ryan
On Fri, Aug 6, 2021 at 1:34 PM Otto Fowler <ot...@gmail.com> wrote:
> You don’t have to assign it to yourself. User submitted jiras are very
> important, because they include use cases and details that may not occur to
> a developer who just picks it up. It is a great way to contribute!
>
> From: Ryan Hendrickson <ry...@gmail.com>
> <ry...@gmail.com>
> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> Date: August 6, 2021 at 03:23:43
> To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
>
> If I put Jira's in, then I'd have to admit I'm a developer vs a user....
> :-)
>
> I'm just a power user at heart :-) ...
>
> Ryan
>
> On Thu, Aug 5, 2021 at 2:01 PM Otto Fowler <ot...@gmail.com>
> wrote:
>
> > Ryan, these are awesome, are there jiras for them? It would be best to
> > capture the requirements / use cases
> >
> > From: Ryan Hendrickson <ry...@gmail.com>
> > <ry...@gmail.com>
> > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > Date: August 4, 2021 at 01:14:23
> > To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> >
> > I'll preface with what I'm about to say is not rooted in any real
> > understanding of the lift required or other customer experiences... We
> > have used NiFi for a long time and are excited for what new features may
> > lay ahead in the 2.x line.
> >
> > ---After having written this, I think our succinct pain points focus
> > around large data flow management, auto-scaling container environments,
> and
> > integration of external analytic/enrichment creation.---
> >
> > Thanks,
> > Ryan
> >
> >
> > 1 - Integration with auto-scaling container environments
> > We often struggle with scaling custom processors in large flows. We've
> > implemented a recent design pattern where we write custom python code
> > packaged into a docker image, fronted with a web-service, and deployed
> into
> > a container environment. In NiFi we use InvokeHTTP to call out to the
> > web-service. If the load becomes too great, we can easily spin-up more
> > than one external web-service without sacrificing the NiFi's overall
> > thruput and thread contention. In this scenario, we're using NiFi to
> > manage the overall dataflow, routes, ETL services, etc, but not so much
> > smarts around auto-scaling processing, etc.
> >
> > This makes it really easy to receive a dataflow analytic/enrichment from
> an
> > external team, wrap it in a docker image, and deploy it into an
> > auto-scaling container environment. This also lets us avoid conversations
> > about how performant the code is and if it's meeting a minimum throughput
> > requirement - typically XXGB/5minutes.
> >
> > The request for 2.x would be to make interacting with container deployed
> > services a more seamless, streamlined process, possibly through a
> > normalized API or as a "Remote Container Environment" that can bind to a
> > service with a health endpoint and verify it's alive and well, etc.
> >
> > 2 - Integration of S3 buckets with Remote Process Groups for outage
> events
> > and data overruns
> > Another frequent struggle is the occasional death of a server in a NiFi
> > flow. Example -- Our overall NiFi flow consists of ~15 stand-alone NiFis
> > and 2 NiFi clusters. The stand-alone NiFi's are linearly chained together
> > with Remote Process Groups to perform the overall dataflow goals. If any
> > one of the 15 experience an outage, data begins to backup on the previous
> > NiFi, hitting backpressure limits, etc. When the down server comes back
> > online, it receives a flood of data, often overwhelming it, necessitating
> > careful flowfile limits and backpressure points.
> >
> > The design pattern we're exploring, but would like to see baked into
> NiFi,
> > is that when a Remote Process Group becomes unavailable due to
> > server-to-server comms failures AND the backpressure limit is reached on
> a
> > relationship, the Remote Process Group would begin sending data to an S3
> > bucket as an automatic failover procedure. When the downed server comes
> > back-online, the Remote Process Group port would know to pull data from
> S3
> > AND receive data via the port from the chained server.
> >
> > 3 - Large Canvas Management
> > Beginning with a NiFi flow that consists of ~15 stand-alone NiFis and 2
> > NiFi clusters, each canvas is managed individually. To see the entire
> > canvas, systems engineering diagrams need to be kept up-to-date
> identifying
> > all the input/output ports. To move processors from server to server
> often
> > requires the verification of the existence of scripts, configuration
> files,
> > NiFi custom nars, and using the templating or Registry to migrate
> portions
> > of the dataflow.
> >
> > Our current design pattern to accomplish some of these tasks include
> using
> > Ansible, Jenkins, and Nexus to verify script & configuration files are up
> > to date. We use Confluence's Gliffy to document the NiFi communication
> > paths between the servers.
> >
> > Ideally, we'd be able to see an overall management view of the Canvases
> > together in a single place - it would indicate which server the portions
> of
> > the canvas are deployed on and tie together where the Remote Process
> Group
> > ports are connected. Additionally, NiFi would allow the user to move
> > Processors and Process Groups seamlessly from one server to another
> server
> > to help manually distribute and balance data flow processing cycles.
> >
> >
> >
> > On Tue, Aug 3, 2021 at 12:45 PM Joe Witt <jo...@gmail.com> wrote:
> >
> > > Kevin,
> > >
> > > This may well be a very interesting approach which opens up a lot of
> > > options for us in how we tackle a 2.x...
> > >
> > > Thanks
> > >
> > > On Tue, Aug 3, 2021 at 8:45 AM Kevin Doran <kd...@gmail.com>
> > > wrote:
> > > >
> > > > Outside of core NiFi, for MiNiFi, I think it would be beneficial to
> > > align MiNiFi Java and MiNiFI C++ to use the same method(s) for command
> > and
> > > control / flow deployment. This was discussed when MiNiFi Java was
> merged
> > > into the NiFi codebase [1].
> > > >
> > > > My proposal would be to create a Java implementation of the C2
> Protocol
> > > used by MiNiFi C++, both server-side and client-side, to allow
> deploying
> > to
> > > MiNiFi Java instances from a centralize flow definition. Thanks to the
> > > unification of MiNiFi Java and NiFi, this could likely also be added as
> a
> > > capability of NiFi at that same time.
> > > >
> > > > If we are able to do this on the 1.x line, then I would also support
> > > deprecating other methods of flow deployment to MiNiFi outlined in [1]
> > and
> > > removing them in a 2.x release.
> > > >
> > > > [1] https://github.com/apache/nifi/pull/4933#issuecomment-808411977
> <
> > > https://github.com/apache/nifi/pull/4933#issuecomment-808411977>
> > > > [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design <
> > > https://cwiki.apache.org/confluence/display/MINIFI/C2+Design>
> > > >
> > > > Thanks,
> > > > Kevin
> > > >
> > > > > On Aug 3, 2021, at 10:07 AM, David Handermann <
> > > exceptionfactory@gmail.com> wrote:
> > > > >
> > > > > Thanks for the continued discussion around what can or should be
> > > removed.
> > > > > Having a clear migration path from version 1 to version 2 will be
> > very
> > > > > important for supporting current deployments.
> > > > >
> > > > > Following the example of some other projects, one way to consider
> the
> > > > > upgrade is from the point of view of deprecation warnings. If
> > > implemented
> > > > > correctly, an installation of NiFi running the latest minor release
> > of
> > > > > version 1, and showing no deprecation warnings, should be
> upgradeable
> > > to
> > > > > version 2 without any changes. Making this work involves accurately
> > > tagging
> > > > > components and methods with deprecation indicators, and providing a
> > > clear
> > > > > way to observe deprecation warnings. With the current version
> > > containing
> > > > > both deprecated components and deprecated methods in various
> classes,
> > > this
> > > > > would involve some work to get right. A simple approach might be a
> > > named
> > > > > logger for deprecation warnings, which would write a separate log
> > > file. A
> > > > > more advanced approach might involve a tool to analyze the system
> and
> > > flow
> > > > > configurations. Either way, it seems that some additional work in a
> > > minor
> > > > > release version is necessary before considering a major release
> > > version.
> > > > >
> > > > > One other point on compatibility: as long as the core component API
> > > > > definition remains backwards compatible, it should be possible to
> run
> > > older
> > > > > components. This provides a transition option for customers using
> > > > > components such as PostHTTP, or any others that are not actively
> > > > > maintained. At some point it may become necessary to break
> > > compatibility at
> > > > > that level, but at least for version 2, maintaining component API
> > > > > compatibility is key.
> > > > >
> > > > > Regards,
> > > > > David Handermann
> > > > >
> > > > > On Sun, Aug 1, 2021 at 10:23 AM Mark Bean <ma...@gmail.com>
> > > wrote:
> > > > >
> > > > >> I created a JIRA ticket to investigate and improve the performance
> > of
> > > > >> InvokeHTTP. It includes a flow definition for benchmarking the
> > > performance
> > > > >> of both PostHTTP and InvokeHTTP.
> > > > >>
> > > > >> https://issues.apache.org/jira/browse/NIFI-8968
> > > > >>
> > > > >> Thanks,
> > > > >> Mark
> > > > >>
> > > > >> On Fri, Jul 30, 2021 at 6:41 PM Adam Taft <ad...@adamtaft.com>
> > wrote:
> > > > >>
> > > > >>> I'm not seeing the side thread that was going to discuss
> > deprecation
> > > of
> > > > >>> PostHTTP. Has that thread started and I just don't see it?
> > > > >>>
> > > > >>> One (significant?) concern with PostHTTP is the smooth
> integration
> > of
> > > > >>> NiFi-to-NiFi communication that is very transparently enabled
> with
> > > the
> > > > >>> ListenHTTP and PostHTTP processors. There's some special logic in
> > > there
> > > > >> for
> > > > >>> handling flowfiles that InvokeHTTP doesn't really (nor should
> > really)
> > > > >> have.
> > > > >>>
> > > > >>> I know of several (large) NiFi installations that rely on the
> > > PostHTTP /
> > > > >>> ListenHTTP combination. It has enabled NiFi to NiFi communication
> > for
> > > > >> folks
> > > > >>> reluctant or unable to enable site-to-site type configuration.
> > > > >>>
> > > > >>> Honestly, I don't know why we'd want to "deprecate" any
> processor,
> > as
> > > > >>> opposed to just moving it to a new location. If these processors
> > can
> > > be
> > > > >>> ported and maintained to whatever the 2.0 API looks like, there's
> > > > >> possibly
> > > > >>> little harm keeping them around.
> > > > >>>
> > > > >>> And by the way, what happened to the "marketplace" concept? Is
> this
> > > being
> > > > >>> considered for 2.0 as well? Because relocating the deprecated
> > > processors
> > > > >>> to an external nar might be the best solution. Losing PostHTTP
> > > entirely I
> > > > >>> think would be a mistake, but I'd conceptually support relocating
> > it.
> > > > >>>
> > > > >>> Thanks,
> > > > >>>
> > > > >>> /Adam
> > > > >>>
> > > > >>> On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <jo...@gmail.com>
> > wrote:
> > > > >>>
> > > > >>>> Looks like we just need to knock out 5 JIRAs :) [1]
> > > > >>>>
> > > > >>>> I felt like we had a label folks were using at one point but
> > quickly
> > > > >>>> looking revealed nothing exciting. After this confluence page
> > > > >>>> stabilizes a bit we can probably knock out some JIRAs and such.
> > > > >>>>
> > > > >>>> [1]
> > https://issues.apache.org/jira/projects/NIFI/versions/12339599
> > > > >>>>
> > > > >>>> On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <
> > > ottobackwards@gmail.com>
> > > > >>>> wrote:
> > > > >>>>>
> > > > >>>>> I find myself wishing I had a list of all the jiras / issues
> that
> > > > >> have
> > > > >>>>> been put off for a 2.0 release because they required some
> change
> > or
> > > > >>>> another
> > > > >>>>> :(
> > > > >>>>>
> > > > >>>>> From: Joe Witt <jo...@gmail.com> <jo...@gmail.com>
> > > > >>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > >> dev@nifi.apache.org>
> > > > >>>>> Date: July 27, 2021 at 12:30:35
> > > > >>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
> > dev@nifi.apache.org
> > > >
> > > > >>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > > >>>>>
> > > > >>>>> A few thoughts:
> > > > >>>>>
> > > > >>>>> 1. I would love to see deprecation notices show up in the UI in
> > > > >>>>> various ways to help motivate users to move off things to more
> > > > >>>>> supportable things. That is not a prerequisite for anything
> > > happening
> > > > >>>>> however. Just a good feature/nice thing to do for users when
> > > someone
> > > > >>>>> is able to tackle it.
> > > > >>>>>
> > > > >>>>> 2. The decision to deprecate something and to further remove it
> > > need
> > > > >>>>> not mean there is a superior solution available. If that thing
> > > itself
> > > > >>>>> isn't getting the love/attention it needs to be
> > > > >>>>> maintained/supported/healthy going forward that alone is enough
> > to
> > > > >>>>> remove it. That might well be the case with PostHTTP [1] and
> for
> > > > >>>>> comparison you can see how much effort has gone into InvokeHTTP
> > > [2].
> > > > >>>>>
> > > > >>>>> 3. When discussing a 2.0 release each thing we add as a 'must
> do'
> > > the
> > > > >>>>> further away from reality such a release will become. We'll
> have
> > to
> > > > >>>>> get very specific about 'musts' vs 'wants'.
> > > > >>>>>
> > > > >>>>> [1]
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > >
> >
> >
>
> https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
> > > > >>>>> [2]
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > >
> >
> >
>
> https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
> > > > >>>>>
> > > > >>>>> On Tue, Jul 27, 2021 at 8:47 AM David Handermann
> > > > >>>>> <ex...@apache.org> wrote:
> > > > >>>>>>
> > > > >>>>>> Thanks Mark, providing a template or comparison statistics
> with
> > > > >> Java
> > > > >>>>>> versions and component configuration details would be very
> > > helpful.
> > > > >>> If
> > > > >>>> it
> > > > >>>>>> is possible to run tests using a public API or deployable
> > service,
> > > > >>> that
> > > > >>>>>> would also help confirm potential differences.
> > > > >>>>>>
> > > > >>>>>> Showing a deprecation notice in the UI could be helpful,
> perhaps
> > > > >> as a
> > > > >>>>>> configurable option. NIFI-8650 describes a general Flow
> Analysis
> > > > >>>>>> capability, and it seems like that might be one possible way
> to
> > > > >>> surface
> > > > >>>>>> deprecation warnings. For something more specific to component
> > > > >>>>> deprecation,
> > > > >>>>>> it seems important to find a balance between making it obvious
> > and
> > > > >>>> making
> > > > >>>>>> it something that ends up getting ignored.
> > > > >>>>>>
> > > > >>>>>> Regards,
> > > > >>>>>> David Handermann
> > > > >>>>>>
> > > > >>>>>> On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <
> > mark.o.bean@gmail.com
> > > >
> > > > >>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> I'll start a new thread for PostHTTP when I get a template
> > and/or
> > > > >>>>> detailed
> > > > >>>>>>> stats.
> > > > >>>>>>>
> > > > >>>>>>> I know the deprecation is noted in the documentation. That's
> a
> > > > >>>>> necessary
> > > > >>>>>>> and minimum level of notification. I was suggesting it be
> more
> > > > >>>> obvious
> > > > >>>>> in
> > > > >>>>>>> the UI. I think it would be beneficial to somehow be aware of
> > the
> > > > >>>>>>> deprecation status simply by looking at the flow (perhaps
> even
> > on
> > > > >>> the
> > > > >>>>>>> summary pages too), and without having to open the
> > documentation
> > > > >>> for
> > > > >>>>> every
> > > > >>>>>>> processor to confirm whether or not a component is marked as
> > > > >>>>> deprecated.
> > > > >>>>>>>
> > > > >>>>>>> Thanks,
> > > > >>>>>>> Mark
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
> > > > >>>>>>> exceptionfactory@apache.org> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> Mark,
> > > > >>>>>>>>
> > > > >>>>>>>> Thanks for the feedback. It may be better to start a
> separate
> > > > >>>> thread
> > > > >>>>> on
> > > > >>>>>>>> PostHTTP, but can you provide an example flow demonstrating
> > the
> > > > >>>>>>> performance
> > > > >>>>>>>> differences between PostHTTP and InvokeHTTP?
> > > > >>>>>>>>
> > > > >>>>>>>> PostHTTP uses the Apache HttpComponents library, whereas
> > > > >>> InvokeHTTP
> > > > >>>>> uses
> > > > >>>>>>>> OkHttp. NiFi 1.13.2 and 1.14.0 included major version
> updates
> > > > >> for
> > > > >>>>> OkHttp,
> > > > >>>>>>>> so it would be important to test with the most recent
> version.
> > > > >> It
> > > > >>>> is
> > > > >>>>> also
> > > > >>>>>>>> worth noting that test classes for PostHTTP are now
> > integration
> > > > >>>> tests
> > > > >>>>> as
> > > > >>>>>>>> opposed to unit tests, which means they are not executed as
> > > > >> part
> > > > >>> of
> > > > >>>>> the
> > > > >>>>>>>> automated builds.
> > > > >>>>>>>>
> > > > >>>>>>>> The NiFi documentation includes the contents of the
> > > > >>>> DeprecationNotice
> > > > >>>>> for
> > > > >>>>>>>> PostHTTP:
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > >
> >
> >
>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
> > > > >>>>>>>>
> > > > >>>>>>>> Regards,
> > > > >>>>>>>> David Handermann
> > > > >>>>>>>>
> > > > >>>>>>>> On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <
> > > > >> mark.o.bean@gmail.com
> > > > >>>>
> > > > >>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> I'm strongly in favor of reducing tech debt, and moving
> > > > >>>>> deliberately
> > > > >>>>>>> to a
> > > > >>>>>>>>> 2.0 release. I have a concern with just one processor that
> is
> > > > >>>>> currently
> > > > >>>>>>>>> marked as deprecated: PostHTTP. (I have not evaluated
> > > > >>>> specifically
> > > > >>>>> any
> > > > >>>>>>>>> other deprecated components; I cannot say if there are or
> are
> > > > >>> not
> > > > >>>>>>> similar
> > > > >>>>>>>>> issues elsewhere.) I understand the rationale for
> deprecating
> > > > >>>> this
> > > > >>>>>>>>> processor in that it eliminates a processor whose
> > > > >> functionality
> > > > >>>> is
> > > > >>>>>>>>> available elsewhere, specifically in the more flexible
> > > > >>>> InvokeHTTP.
> > > > >>>>>>>> However,
> > > > >>>>>>>>> in my experience and testing, PostHTTP performs transfers
> > > > >> with
> > > > >>>> far
> > > > >>>>>>>> greater
> > > > >>>>>>>>> throughput than InvokeHTTP. I would not be in favor of
> > > > >> removing
> > > > >>>>>>> PostHTTP
> > > > >>>>>>>>> unless/until InvokeHTTP is refactored to increase its
> > > > >>> throughput
> > > > >>>>>>>>> capability.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Has anyone else continued to use PostHTTP over InvokeHTTP
> for
> > > > >>>>> similar
> > > > >>>>>>>>> reasons? Or, is there a performance-related configuration
> for
> > > > >>>>>>> InvokeHTTP
> > > > >>>>>>>> I
> > > > >>>>>>>>> may have missed?
> > > > >>>>>>>>>
> > > > >>>>>>>>> Also, in order to help facilitate a smooth transition to
> 2.0
> > > > >>>> from a
> > > > >>>>>>> user
> > > > >>>>>>>>> perspective, would it be advisable to add some sort of
> visual
> > > > >>>>> indicator
> > > > >>>>>>>> in
> > > > >>>>>>>>> the UI for components that are currently annotated as
> > > > >>>> @Deprecated?
> > > > >>>>> This
> > > > >>>>>>>>> might help users proactively modify their flow prior to a
> > > > >>> release
> > > > >>>>> that
> > > > >>>>>>>>> would otherwise break it.
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <
> > > > >> bbende@gmail.com>
> > > > >>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> With the merging of MiNiFi and registry into the NiFi
> repo,
> > > > >>>> we've
> > > > >>>>>>>>>> actually gone more towards a mono-repo where everything is
> > > > >>>>> released
> > > > >>>>>>>>>> together. Eventually it would still be nice to have a
> > > > >> smaller
> > > > >>>>> base
> > > > >>>>>>>>>> distribution containing just the framework and standard
> > > > >> NARs,
> > > > >>>> but
> > > > >>>>> it
> > > > >>>>>>>>>> is hard to tackle that until we provide a way for users to
> > > > >>>> easily
> > > > >>>>>>>>>> obtain the NARs in some other way.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <
> > > > >>>>> edward.armes@gmail.com
> > > > >>>>>>>>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Given the major version number shift and the spliting up
> > > > >> of
> > > > >>>>>>>> processors
> > > > >>>>>>>>>> into
> > > > >>>>>>>>>>> multiple NAR's. I'd like to suggest that we start
> > > > >>>> individually
> > > > >>>>>>>>> versioning
> > > > >>>>>>>>>>> individual NARs/ bundles.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I can see this bringing a large number of benifits
> > > > >>> including
> > > > >>>>> making
> > > > >>>>>>>>> Nifi
> > > > >>>>>>>>>>> more deployable with things RPM, but also potentially
> > > > >>>> allowing
> > > > >>>>>>> those
> > > > >>>>>>>>> that
> > > > >>>>>>>>>>> have to deploy managed Nifi instances easier to upgrade.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Edward
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Mon, 26 Jul 2021, 20:42 Otto Fowler, <
> > > > >>>> ottobackwards@gmail.com>
> > > > >>>>>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> The issue with updating the aws sdk is if it breaks any
> > > > >>> one
> > > > >>>>> of
> > > > >>>>>>> the
> > > > >>>>>>>>>>>> processors.
> > > > >>>>>>>>>>>> the Web Gateway API invoke processor for example is not
> > > > >>>> using
> > > > >>>>> a
> > > > >>>>>>>> high
> > > > >>>>>>>>>> level
> > > > >>>>>>>>>>>> purpose build client and may break.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> If we change the aws version, we need to coordinate in
> > > > >>>> such a
> > > > >>>>> way
> > > > >>>>>>>>> that
> > > > >>>>>>>>>> they
> > > > >>>>>>>>>>>> all
> > > > >>>>>>>>>>>> can come along reasonably.
> > > > >>>>>>>>>>>> IE: what happens if 1 or 2 break but the rest or OK?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> From: David Handermann <ex...@apache.org>
> > > > >>>>>>>>>>>> <ex...@apache.org>
> > > > >>>>>>>>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > >>>>>>>>> dev@nifi.apache.org>
> > > > >>>>>>>>>>>> Date: July 26, 2021 at 09:33:42
> > > > >>>>>>>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > >>>>>>> dev@nifi.apache.org
> > > > >>>>>>>>>
> > > > >>>>>>>>>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Chris,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Thanks for the reply and recommendations. It seems like
> > > > >>>> some
> > > > >>>>> of
> > > > >>>>>>> the
> > > > >>>>>>>>>> work to
> > > > >>>>>>>>>>>> reorganize the module structure could be done outside
> > > > >> of
> > > > >>> a
> > > > >>>>> major
> > > > >>>>>>>>>> release,
> > > > >>>>>>>>>>>> but it would be great to target any breaking changes
> > > > >> for
> > > > >>>> 2.0.
> > > > >>>>>>>>> Perhaps a
> > > > >>>>>>>>>>>> separate feature proposal on module restructuring, with
> > > > >>> the
> > > > >>>>> goal
> > > > >>>>>>> of
> > > > >>>>>>>>>>>> supporting optimized builds, would be a helpful way to
> > > > >>> move
> > > > >>>>> that
> > > > >>>>>>>> part
> > > > >>>>>>>>>> of
> > > > >>>>>>>>>>>> the discussion forward.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Regarding updating AWS SDK to version 2, it seems like
> > > > >>> that
> > > > >>>>> might
> > > > >>>>>>>> be
> > > > >>>>>>>>>>>> possible now. I haven't taken a close look at the
> > > > >>>> referencing
> > > > >>>>>>>>>> components,
> > > > >>>>>>>>>>>> so I'm not sure about the level of effort involved.
> > > > >> Minor
> > > > >>>>> NiFi
> > > > >>>>>>>>> version
> > > > >>>>>>>>>>>> updates have incorporated major new versions of
> > > > >>>> dependencies.
> > > > >>>>> For
> > > > >>>>>>>>>> example,
> > > > >>>>>>>>>>>> NiFi 1.14 included an upgrade from Spring Framework 4
> > > > >> to
> > > > >>> 5.
> > > > >>>>> On
> > > > >>>>>>> the
> > > > >>>>>>>>> one
> > > > >>>>>>>>>>>> hand, including the AWS SDK update as part of a major
> > > > >>>> release
> > > > >>>>>>> seems
> > > > >>>>>>>>>>>> helpful, but unless there are changes that break
> > > > >> existing
> > > > >>>>>>> component
> > > > >>>>>>>>>>>> properties, upgrading the AWS SDK could be worked
> > > > >>>>> independently.
> > > > >>>>>>>>>> Others may
> > > > >>>>>>>>>>>> have more insight into particular usage of that
> > > > >> library.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Regards,
> > > > >>>>>>>>>>>> David Handermann
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > > > >>>>>>>>>>>> <ch...@naimuri.com.invalid> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Might be worth considering refactoring the build as
> > > > >>> part
> > > > >>>> of
> > > > >>>>>>> this
> > > > >>>>>>>>> work
> > > > >>>>>>>>>>>> too,
> > > > >>>>>>>>>>>>> e.g. only building the bits of the repo affected by a
> > > > >>>>> commit,
> > > > >>>>>>>> etc.
> > > > >>>>>>>>> -
> > > > >>>>>>>>>>>>> discussed briefly in previous threads but don't think
> > > > >>> any
> > > > >>>>>>> changes
> > > > >>>>>>>>>> made
> > > > >>>>>>>>>>>> yet.
> > > > >>>>>>>>>>>>> If NARs/components are likely to be split up and
> > > > >>>> refactored
> > > > >>>>>>> then
> > > > >>>>>>>>> such
> > > > >>>>>>>>>>>> work
> > > > >>>>>>>>>>>>> around the build probably makes sense to consider.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> I've a couple of PRs open that include updates to
> > > > >>>>> Elasticsearch
> > > > >>>>>>>>>> versions
> > > > >>>>>>>>>>>>> already, although I stopped at 7.10.2 (after which
> > > > >>>> Elastic
> > > > >>>>>>>> changed
> > > > >>>>>>>>>>>> licence)
> > > > >>>>>>>>>>>>> in case there were licence concerns. But more work
> > > > >> can
> > > > >>> be
> > > > >>>>> done
> > > > >>>>>>> to
> > > > >>>>>>>>>> tidy up
> > > > >>>>>>>>>>>>> the processors, absolutely.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> AWS libraries to v2 would seem a sensible move and a
> > > > >>>>> refactor
> > > > >>>>>>> of
> > > > >>>>>>>>>> those
> > > > >>>>>>>>>>>>> processors as well.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Cheers,
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Chris Sampson
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > > > >>>>>>>>>>>> exceptionfactory@apache.org>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Thanks for pointing out the standard NAR bundles,
> > > > >>> Mark.
> > > > >>>>> There
> > > > >>>>>>>>> are a
> > > > >>>>>>>>>>>>> number
> > > > >>>>>>>>>>>>>> of components in the standard NAR bundles with
> > > > >>>> particular
> > > > >>>>>>>>>> dependencies
> > > > >>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>> would make more sense in separate NARs.
> > > > >> Reorganizing
> > > > >>>> the
> > > > >>>>>>>> standard
> > > > >>>>>>>>>> NAR
> > > > >>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>> components with limited dependencies and wide
> > > > >>>>> applicability
> > > > >>>>>>>> would
> > > > >>>>>>>>>>>>>> definitely help with future maintenance.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Regards,
> > > > >>>>>>>>>>>>>> David Handermann
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> > > > >>>>>>>>> markap14@hotmail.com>
> > > > >>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> There’s also some code that exists in order to
> > > > >>>> maintain
> > > > >>>>>>>>> backward
> > > > >>>>>>>>>>>>>>> compatibility in the repositories. I would very
> > > > >>> much
> > > > >>>>> like
> > > > >>>>>>> the
> > > > >>>>>>>>>>>>>> repositories
> > > > >>>>>>>>>>>>>>> to contain no unnecessary code. And swap file
> > > > >>> format
> > > > >>>>>>> supports
> > > > >>>>>>>>>> really
> > > > >>>>>>>>>>>>> old
> > > > >>>>>>>>>>>>>>> formats. And the old impls of the repositories
> > > > >>>>> themselves,
> > > > >>>>>>>> like
> > > > >>>>>>>>>>>>>>> PersistentProvRepo instead of WriteAheadProv
> > > > >> Repo,
> > > > >>>> etc.
> > > > >>>>>>> Lots
> > > > >>>>>>>> of
> > > > >>>>>>>>>> stuff
> > > > >>>>>>>>>>>>>> there
> > > > >>>>>>>>>>>>>>> that can be removed. And some methods in
> > > > >>>> ProcessSession
> > > > >>>>>>> that
> > > > >>>>>>>>> are
> > > > >>>>>>>>>>>> never
> > > > >>>>>>>>>>>>>> used
> > > > >>>>>>>>>>>>>>> by any processor in the codebase but exists in
> > > > >> the
> > > > >>>>> public
> > > > >>>>>>> API
> > > > >>>>>>>>> so
> > > > >>>>>>>>>>>> can’t
> > > > >>>>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>> removed till 2.0.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> I think his is also a great time to clean up the
> > > > >>>>> “standard
> > > > >>>>>>>>> nar.”
> > > > >>>>>>>>>> At
> > > > >>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>> point, it’s something like 70 MB. And many of the
> > > > >>>>>>> components
> > > > >>>>>>>>>> there
> > > > >>>>>>>>>>>> are
> > > > >>>>>>>>>>>>>> not
> > > > >>>>>>>>>>>>>>> really “standard” - things like connecting to
> > > > >> FTP &
> > > > >>>>> SFTP
> > > > >>>>>>>>>> servers, XML
> > > > >>>>>>>>>>>>>>> processing, Jolt transform, etc. could
> > > > >> potentially
> > > > >>> be
> > > > >>>>> moved
> > > > >>>>>>>>> into
> > > > >>>>>>>>>>>> other
> > > > >>>>>>>>>>>>>>> nars. The
> > > > >>>>> nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
> > > > >>>>>>> is
> > > > >>>>>>>>>> 6.9 MB
> > > > >>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>> not
> > > > >>>>>>>>>>>>>>> necessary for stateless or minifi java. Lots of
> > > > >>>> things
> > > > >>>>>>>> probably
> > > > >>>>>>>>>> to
> > > > >>>>>>>>>>>>>>> reconsider within the standard nar.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> I definitely think this is a reasonable approach,
> > > > >>> to
> > > > >>>>> allow
> > > > >>>>>>>> for
> > > > >>>>>>>>> a
> > > > >>>>>>>>>> 2.0
> > > > >>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>> is not a huge feature release but allows the
> > > > >>> project
> > > > >>>> to
> > > > >>>>> be
> > > > >>>>>>>>>> simpler
> > > > >>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>> more
> > > > >>>>>>>>>>>>>>> nimble.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Thanks
> > > > >>>>>>>>>>>>>>> -Mark
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > > > >>>>>>>>>> mikerthomsen@gmail.com
> > > > >>>>>>>>>>>>>> <mailto:
> > > > >>>>>>>>>>>>>>> mikerthomsen@gmail.com>> wrote:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Russell,
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> AFAICT from looking at Elastic's repos, the low
> > > > >>> level
> > > > >>>>> REST
> > > > >>>>>>>>>> client is
> > > > >>>>>>>>>>>>>>> still fine.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > >
> >
> >
>
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Our Elasticsearch support is spread over two NARs
> > > > >>> at
> > > > >>>>>>> present.
> > > > >>>>>>>>> One
> > > > >>>>>>>>>>>> uses
> > > > >>>>>>>>>>>>>>> OkHttp and the other uses that low level Elastic
> > > > >>> REST
> > > > >>>>>>> client.
> > > > >>>>>>>>>>>>>>> Therefore, I think we're fine on licensing for
> > > > >> the
> > > > >>>>> moment.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Mike
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > > > >>>>>>>>>>>> russ@windofkeltia.com
> > > > >>>>>>>>>>>>>>> <ma...@windofkeltia.com>> wrote:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Bringing up Elastic also reminds me that the
> > > > >>> Elastic
> > > > >>>>>>>> framework
> > > > >>>>>>>>>> has
> > > > >>>>>>>>>>>> just
> > > > >>>>>>>>>>>>>>> recently transitioned out of Open Source, so to
> > > > >>>>> acknowledge
> > > > >>>>>>>>> that,
> > > > >>>>>>>>>>>> maybe
> > > > >>>>>>>>>>>>>>> some effort toward OpenSearch--I say this not
> > > > >>>>> understanding
> > > > >>>>>>>>>> exactly
> > > > >>>>>>>>>>>> how
> > > > >>>>>>>>>>>>>>> this sort of thing is considered in a
> > > > >> large-scale,
> > > > >>>>>>>> world-class
> > > > >>>>>>>>>>>> software
> > > > >>>>>>>>>>>>>>> project like Apache NiFi. (I'm not a contributor,
> > > > >>>> just
> > > > >>>>> a
> > > > >>>>>>>>> grateful
> > > > >>>>>>>>>>>>>>> consumer.)
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Russ
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > > >>>>>>>>>>>>>>> Along with the itemized list for ancient
> > > > >> components
> > > > >>>> we
> > > > >>>>>>> should
> > > > >>>>>>>>>> look at
> > > > >>>>>>>>>>>>>>> updating versions of drivers, SDKs, etc. for
> > > > >>> external
> > > > >>>>>>> systems
> > > > >>>>>>>>>> such as
> > > > >>>>>>>>>>>>>>> Elasticsearch, Cassandra, etc. There may be
> > > > >>> breaking
> > > > >>>>>>> changes
> > > > >>>>>>>>> but
> > > > >>>>>>>>>> 2.0
> > > > >>>>>>>>>>>>>>> is probably the right time to get things up to
> > > > >> date
> > > > >>>> to
> > > > >>>>> make
> > > > >>>>>>>>> them
> > > > >>>>>>>>>> more
> > > > >>>>>>>>>>>>>>> useful to more people.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > > > >>>>>>>>>> thenatog@gmail.com
> > > > >>>>>>>>>>>>>> <mailto:
> > > > >>>>>>>>>>>>>>> thenatog@gmail.com>> wrote:
> > > > >>>>>>>>>>>>>>> I'm a +1 for removing pretty much all of this
> > > > >>> stuff.
> > > > >>>>> There
> > > > >>>>>>>> are
> > > > >>>>>>>>>>>> security
> > > > >>>>>>>>>>>>>>> implications to keeping old dependencies around,
> > > > >> so
> > > > >>>> the
> > > > >>>>>>> more
> > > > >>>>>>>>> old
> > > > >>>>>>>>>> code
> > > > >>>>>>>>>>>>> we
> > > > >>>>>>>>>>>>>>> can remove the better. I agree that eventually we
> > > > >>>> need
> > > > >>>>> to
> > > > >>>>>>>> move
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>>>>> supporting only Java 11+, and as our next release
> > > > >>>> will
> > > > >>>>>>>> probably
> > > > >>>>>>>>>> be
> > > > >>>>>>>>>>>>> about
> > > > >>>>>>>>>>>>>> 4
> > > > >>>>>>>>>>>>>>> - 6 months from now that doesn't seem too soon.
> > > > >> We
> > > > >>>>> could
> > > > >>>>>>>>>> potentially
> > > > >>>>>>>>>>>>>> break
> > > > >>>>>>>>>>>>>>> this in two and remove the deprecated processors
> > > > >>> and
> > > > >>>>> leave
> > > > >>>>>>>> 1.x
> > > > >>>>>>>>> on
> > > > >>>>>>>>>>>> Java
> > > > >>>>>>>>>>>>> 8,
> > > > >>>>>>>>>>>>>>> and finally start on 2.x which would support only
> > > > >>>> Java
> > > > >>>>> 11.
> > > > >>>>>>>> I'm
> > > > >>>>>>>>>> unsure
> > > > >>>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>>> what implications changing the date and time
> > > > >>> handling
> > > > >>>>> would
> > > > >>>>>>>>> have
> > > > >>>>>>>>>> -
> > > > >>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>> running systems that use long term historical
> > > > >> logs,
> > > > >>>>>>>> unexpected
> > > > >>>>>>>>>>>> impacts
> > > > >>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>> time logging could be a problem.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> As Joe says I think feature work will have to be
> > > > >>>>> dedicated
> > > > >>>>>>> to
> > > > >>>>>>>>>> 2.x and
> > > > >>>>>>>>>>>>> we
> > > > >>>>>>>>>>>>>>> could support 1.x for security fixes for some
> > > > >>> period
> > > > >>>> of
> > > > >>>>>>> time.
> > > > >>>>>>>>> 2.x
> > > > >>>>>>>>>>>> seems
> > > > >>>>>>>>>>>>>>> like a gargantuan task but it's probably time to
> > > > >>> get
> > > > >>>>>>> started.
> > > > >>>>>>>>> Not
> > > > >>>>>>>>>>>> sure
> > > > >>>>>>>>>>>>>> how
> > > > >>>>>>>>>>>>>>> we handle all open PRs and the transition between
> > > > >>> 1.x
> > > > >>>>> and
> > > > >>>>>>>> 2.x.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> > > > >>>>>>>> joe.witt@gmail.com
> > > > >>>>>>>>>>>> <mailto:
> > > > >>>>>>>>>>>>>>> joe.witt@gmail.com>> wrote:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Jon
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> You're right we have to be careful and you're
> > > > >> right
> > > > >>>>> there
> > > > >>>>>>> are
> > > > >>>>>>>>>> still
> > > > >>>>>>>>>>>>>>> significant Java 8 users out there. But we also
> > > > >>> have
> > > > >>>> to
> > > > >>>>> be
> > > > >>>>>>>>>> careful
> > > > >>>>>>>>>>>>>>> about security and sustainability of the
> > > > >> codebase.
> > > > >>> If
> > > > >>>>> we
> > > > >>>>>>> had
> > > > >>>>>>>>>> talked
> > > > >>>>>>>>>>>>>>> about this last year when that article came out
> > > > >> I'd
> > > > >>>>> have
> > > > >>>>>>>> agreed
> > > > >>>>>>>>>> it is
> > > > >>>>>>>>>>>>>>> too early. Interestingly that link seems to get
> > > > >>>> updated
> > > > >>>>>>> and I
> > > > >>>>>>>>>> tried
> > > > >>>>>>>>>>>>>>> [1] and found more recent data (not sure how
> > > > >>> recent).
> > > > >>>>>>> Anyway
> > > > >>>>>>>> it
> > > > >>>>>>>>>>>>>>> suggests Java 8 is still the top dog but we see
> > > > >>> good
> > > > >>>>> growth
> > > > >>>>>>>> on
> > > > >>>>>>>>>> 11. In
> > > > >>>>>>>>>>>>>>> my $dayjob this aligns to what I'm seeing too.
> > > > >>>>> Customers
> > > > >>>>>>>> didn't
> > > > >>>>>>>>>> seem
> > > > >>>>>>>>>>>>>>> to care about Java 11 until later half last year
> > > > >>> and
> > > > >>>>> now
> > > > >>>>>>>>>> suddenly it
> > > > >>>>>>>>>>>>>>> is all over the place.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> I think once we put out a NiFi 2.0 release we'd
> > > > >> see
> > > > >>>>> rapid
> > > > >>>>>>>>>> decrease in
> > > > >>>>>>>>>>>>>>> work on the 1.x line just being blunt. We did
> > > > >> this
> > > > >>>> many
> > > > >>>>>>> years
> > > > >>>>>>>>> ago
> > > > >>>>>>>>>>>>>>> with 0.x to 1.x and we stood behind 0.x for a
> > > > >> while
> > > > >>>>> (maybe
> > > > >>>>>>> a
> > > > >>>>>>>>>> year or
> > > > >>>>>>>>>>>>>>> so) but it was purely bug fix/security related
> > > > >>> bits.
> > > > >>>> We
> > > > >>>>>>> would
> > > > >>>>>>>>>> need to
> > > > >>>>>>>>>>>>>>> do something similar. But feature work would
> > > > >> almost
> > > > >>>>>>> certainly
> > > > >>>>>>>>> go
> > > > >>>>>>>>>> to
> > > > >>>>>>>>>>>>>>> the 2.x line. Maybe there are other workable
> > > > >> models
> > > > >>>> but
> > > > >>>>> my
> > > > >>>>>>>>>> instinct
> > > > >>>>>>>>>>>>>>> suggests this is likely to follow a similar path.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> ...anyway I agree it isn't that easy of a call to
> > > > >>>> dump
> > > > >>>>> Java
> > > > >>>>>>>> 8.
> > > > >>>>>>>>> We
> > > > >>>>>>>>>>>>>>> need to make the call in both the interests of
> > > > >> the
> > > > >>>> user
> > > > >>>>>>> base
> > > > >>>>>>>>> and
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>>>>>> contributor base of the community.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> [1]
> > > > >>>> https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Thanks
> > > > >>>>>>>>>>>>>>> Joe
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
> > > > >>>>>>> joe.witt@gmail.com
> > > > >>>>>>>>>> <mailto:
> > > > >>>>>>>>>>>>>>> joe.witt@gmail.com>> wrote:
> > > > >>>>>>>>>>>>>>> Russ
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Yeah the flow registry is a key part of it. But
> > > > >>> also
> > > > >>>>> now
> > > > >>>>>>> you
> > > > >>>>>>>>> can
> > > > >>>>>>>>>>>>>>> download the flow definition in JSON (upload i
> > > > >>> think
> > > > >>>> is
> > > > >>>>>>> there
> > > > >>>>>>>>> now
> > > > >>>>>>>>>>>>>>> too). Templates offered a series of challenges
> > > > >> such
> > > > >>>> as
> > > > >>>>> we
> > > > >>>>>>>> store
> > > > >>>>>>>>>> them
> > > > >>>>>>>>>>>>>>> in the flow definition which has made flows
> > > > >> massive
> > > > >>>> in
> > > > >>>>> an
> > > > >>>>>>>>>> unintended
> > > > >>>>>>>>>>>>>>> way which isn't fun for cluster behavior.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> We have a couple cases where we headed down a
> > > > >>>>> particular
> > > > >>>>>>>>> concept
> > > > >>>>>>>>>> and
> > > > >>>>>>>>>>>>>>> came up with better approaches later. We need to
> > > > >>>>> reconcile
> > > > >>>>>>>>> these
> > > > >>>>>>>>>> with
> > > > >>>>>>>>>>>>>>> the benefit of hindsight, and while being careful
> > > > >>> to
> > > > >>>> be
> > > > >>>>> not
> > > > >>>>>>>>>> overly
> > > > >>>>>>>>>>>>>>> disruptive to existing users, to reduce the
> > > > >>>>>>>>> codebase/maintenance
> > > > >>>>>>>>>>>>>>> burden and allow continued evolution of the
> > > > >>> project.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Thanks
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > > > >>>>>>>>>>>> russ@windofkeltia.com
> > > > >>>>>>>>>>>>>>> <ma...@windofkeltia.com>>
> > > > >>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>> Joe,
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> I apologize for the off-topic intrusion, but what
> > > > >>>>> replaces
> > > > >>>>>>>>>> templates?
> > > > >>>>>>>>>>>>>>> The Registry? Templates rocked and we have used
> > > > >>> them
> > > > >>>>> since
> > > > >>>>>>>>> 0.5.x.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Russ
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > >>>>>>>>>>>>>>> David,
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> I think this is a highly reasonable approach and
> > > > >>>> such a
> > > > >>>>>>> focus
> > > > >>>>>>>>>> will
> > > > >>>>>>>>>>>>>>> greatly help make a 2.0 release far more
> > > > >>> approachable
> > > > >>>>> to
> > > > >>>>>>>> knock
> > > > >>>>>>>>>> out.
> > > > >>>>>>>>>>>>>>> Not only that but tech debt reduction would help
> > > > >>> make
> > > > >>>>> work
> > > > >>>>>>>>>> towards
> > > > >>>>>>>>>>>>>>> major features we'd think about in a 'major
> > > > >>> release'
> > > > >>>>> sense
> > > > >>>>>>>> more
> > > > >>>>>>>>>>>>>>> approachable.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> We should remove all deprecated things (as well
> > > > >> as
> > > > >>>>> verify
> > > > >>>>>>> we
> > > > >>>>>>>>>> have the
> > > > >>>>>>>>>>>>>>> right list). We should remove/consider removal of
> > > > >>>>>>> deprecated
> > > > >>>>>>>>>>>>>>> concepts
> > > > >>>>>>>>>>>>>>> like templates. We should consider whether we can
> > > > >>>>> resolve
> > > > >>>>>>> the
> > > > >>>>>>>>>>>>>>> various
> > > > >>>>>>>>>>>>>>> ways we've handled what are now parameters down
> > > > >> to
> > > > >>>> one
> > > > >>>>>>> clean
> > > > >>>>>>>>>>>>>>> approach.
> > > > >>>>>>>>>>>>>>> We should remove options in the nifi.properties
> > > > >>> which
> > > > >>>>> turn
> > > > >>>>>>>> out
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>>>>> never be used quite right (if there are). There
> > > > >> is
> > > > >>>>> quite a
> > > > >>>>>>>> bit
> > > > >>>>>>>>> we
> > > > >>>>>>>>>>>>>>> can
> > > > >>>>>>>>>>>>>>> do purely in the name of tech debt reduction.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Lots to consider here but I think this is the
> > > > >> right
> > > > >>>>>>>> discussion.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Than ks
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> > > > >>>>>>>> bbende@gmail.com
> > > > >>>>>>>>>>>> <mailto:
> > > > >>>>>>>>>>>>>>> bbende@gmail.com>>
> > > > >>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>> I'm a +1 for this... Not sure if this falls under
> > > > >>>>> "Removing
> > > > >>>>>>>>>>>>>>> Deprecated
> > > > >>>>>>>>>>>>>>> Components", but I think we should also look at
> > > > >>>>> anything
> > > > >>>>>>> that
> > > > >>>>>>>>> has
> > > > >>>>>>>>>>>>>>> been
> > > > >>>>>>>>>>>>>>> marked as deprecated throughout the code base as
> > > > >> a
> > > > >>>>>>> candidate
> > > > >>>>>>>>> for
> > > > >>>>>>>>>>>>>>> removal. There are quite a few classes, methods,
> > > > >>>>>>> properties,
> > > > >>>>>>>>> etc
> > > > >>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>> have been waiting for a chance to be removed.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > >>>>>>>>>>>>>>> <exceptionfactory@apache.org<mailto:
> > > > >>>>>>>>> exceptionfactory@apache.org
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>> Team,
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> With all of the excellent work that many have
> > > > >>>>> contributed
> > > > >>>>>>> to
> > > > >>>>>>>>> NiFi
> > > > >>>>>>>>>>>>>>> over the
> > > > >>>>>>>>>>>>>>> years, the code base has also accumulated some
> > > > >>> amount
> > > > >>>>> of
> > > > >>>>>>>>>> technical
> > > > >>>>>>>>>>>>>>> debt. A
> > > > >>>>>>>>>>>>>>> handful of components have been marked as
> > > > >>> deprecated,
> > > > >>>>> and
> > > > >>>>>>>> some
> > > > >>>>>>>>>>>>>>> components
> > > > >>>>>>>>>>>>>>> remain in the code base to support integration
> > > > >> with
> > > > >>>> old
> > > > >>>>>>>>> versions
> > > > >>>>>>>>>>>>>>> of various
> > > > >>>>>>>>>>>>>>> products. Following the principles of semantic
> > > > >>>>> versioning,
> > > > >>>>>>>>>>>>>>> introducing a
> > > > >>>>>>>>>>>>>>> major release would provide the opportunity to
> > > > >>> remove
> > > > >>>>> these
> > > > >>>>>>>>>>>>>>> deprecated and
> > > > >>>>>>>>>>>>>>> unsupported components.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Rather than focusing the next major release on
> > > > >> new
> > > > >>>>>>> features,
> > > > >>>>>>>>> what
> > > > >>>>>>>>>>>>>>> do you
> > > > >>>>>>>>>>>>>>> think about focusing on technical debt removal?
> > > > >>> This
> > > > >>>>>>> approach
> > > > >>>>>>>>>>>>>>> would not
> > > > >>>>>>>>>>>>>>> make for the most interesting release, but it
> > > > >>>> provides
> > > > >>>>> the
> > > > >>>>>>>>>>>>>>> opportunity to
> > > > >>>>>>>>>>>>>>> clean up elements that involve breaking changes.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Focusing on technical debt, at least three
> > > > >> primary
> > > > >>>>> goals
> > > > >>>>>>> come
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>>>>> mind for
> > > > >>>>>>>>>>>>>>> the next major release:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> 1. Removal of deprecated and unmaintained
> > > > >>> components
> > > > >>>>>>>>>>>>>>> 2. Require Java 11 as the minimum supported
> > > > >> version
> > > > >>>>>>>>>>>>>>> 3. Transition internal date and time handling to
> > > > >>> JSR
> > > > >>>>> 310
> > > > >>>>>>>>>> java.time
> > > > >>>>>>>>>>>>>>> components
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> *Removing Deprecated Components*
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Removing support for older and deprecated
> > > > >>> components
> > > > >>>>>>>> provides a
> > > > >>>>>>>>>>>>>>> great
> > > > >>>>>>>>>>>>>>> opportunity to improve the overall security
> > > > >> posture
> > > > >>>>> when it
> > > > >>>>>>>>> comes
> > > > >>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>> maintaining dependencies. The OWASP dependency
> > > > >>> plugin
> > > > >>>>>>> report
> > > > >>>>>>>>>>>>>>> currently
> > > > >>>>>>>>>>>>>>> generates 50 MB of HTML for questionable
> > > > >>>> dependencies,
> > > > >>>>> many
> > > > >>>>>>>> of
> > > > >>>>>>>>>>>>>>> which are
> > > > >>>>>>>>>>>>>>> related to old versions of various libraries.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> As a starting point, here are a handful of
> > > > >>> components
> > > > >>>>> and
> > > > >>>>>>>>>>>>>>> extension modules
> > > > >>>>>>>>>>>>>>> that could be targeted for removal in a major
> > > > >>>> version:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> - PostHTTP and GetHTTP
> > > > >>>>>>>>>>>>>>> - ListenLumberjack and the entire
> > > > >>>>> nifi-lumberjack-bundle
> > > > >>>>>>>>>>>>>>> - ListenBeats and the entire nifi-beats-bundle
> > > > >>>>>>>>>>>>>>> - Elasticsearch 5 components
> > > > >>>>>>>>>>>>>>> - Hive 1 and 2 components
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> *Requiring Java 11*
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Java 8 is now over seven years old, and NiFi has
> > > > >>>>> supported
> > > > >>>>>>>>>> general
> > > > >>>>>>>>>>>>>>> compatibility with Java 11 for several years.
> > > > >> NiFi
> > > > >>>>> 1.14.0
> > > > >>>>>>>>>>>>>>> incorporated
> > > > >>>>>>>>>>>>>>> internal improvements specifically related to TLS
> > > > >>>> 1.3,
> > > > >>>>>>> which
> > > > >>>>>>>>>>>>>>> allowed
> > > > >>>>>>>>>>>>>>> closing out the long-running Java 11
> > > > >> compatibility
> > > > >>>> epic
> > > > >>>>>>>>>> NIFI-5174.
> > > > >>>>>>>>>>>>>>> Making
> > > > >>>>>>>>>>>>>>> Java 11 the minimum required version provides the
> > > > >>>>>>> opportunity
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>>>>> address
> > > > >>>>>>>>>>>>>>> any lingering edge cases and put NiFi in a better
> > > > >>>>> position
> > > > >>>>>>> to
> > > > >>>>>>>>>>>>>>> support
> > > > >>>>>>>>>>>>>>> current Java versions.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> *JSR 310 for Date and Time Handling*
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Without making the scope too broad, transitioning
> > > > >>>>> internal
> > > > >>>>>>>> date
> > > > >>>>>>>>>>>>>>> and time
> > > > >>>>>>>>>>>>>>> handling to use DateTimeFormatter instead of
> > > > >>>>>>> SimpleDateFormat
> > > > >>>>>>>>>>>>>>> would provide
> > > > >>>>>>>>>>>>>>> a number of advantages. The Java Time components
> > > > >>>>> provide
> > > > >>>>>>> much
> > > > >>>>>>>>>>>>>>> better
> > > > >>>>>>>>>>>>>>> clarity when it comes to handling localized date
> > > > >>> and
> > > > >>>>> time
> > > > >>>>>>>>>>>>>>> representations,
> > > > >>>>>>>>>>>>>>> and also avoid the inherent confusion of
> > > > >>>> java.sql.Date
> > > > >>>>>>>>> extending
> > > > >>>>>>>>>>>>>>> java.util.Date. Many internal components,
> > > > >>>> specifically
> > > > >>>>>>>>>>>>>>> Record-oriented
> > > > >>>>>>>>>>>>>>> processors and services, rely on date parsing,
> > > > >>>> leading
> > > > >>>>> to
> > > > >>>>>>>>>>>>>>> confusion and
> > > > >>>>>>>>>>>>>>> various workarounds. The pattern formats of
> > > > >>>>>>> SimpleDateFormat
> > > > >>>>>>>>> and
> > > > >>>>>>>>>>>>>>> DateTimeFormatter are very similar, but there
> > > > >> are a
> > > > >>>> few
> > > > >>>>>>>> subtle
> > > > >>>>>>>>>>>>>>> differences.
> > > > >>>>>>>>>>>>>>> Making this transition would provide a much
> > > > >> better
> > > > >>>>>>> foundation
> > > > >>>>>>>>>>>>>>> going forward.
> > > > >>>>>>>>>>>>>>> *Conclusion*
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Thanks for giving this proposal some
> > > > >> consideration.
> > > > >>>>> Many of
> > > > >>>>>>>> you
> > > > >>>>>>>>>>>>>>> have been
> > > > >>>>>>>>>>>>>>> developing NiFi for years and I look forward to
> > > > >>> your
> > > > >>>>>>>> feedback.
> > > > >>>>>>>>> I
> > > > >>>>>>>>>>>>>>> would be
> > > > >>>>>>>>>>>>>>> glad to put together a more formalized
> > > > >>> recommendation
> > > > >>>>> on
> > > > >>>>>>>>>>>>>>> Confluence and
> > > > >>>>>>>>>>>>>>> write up Jira epics if this general approach
> > > > >> sounds
> > > > >>>>>>> agreeable
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>> community.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Regards,
> > > > >>>>>>>>>>>>>>> David Handermann
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> > >
> >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Otto Fowler <ot...@gmail.com>.
You don’t have to assign it to yourself. User submitted jiras are very
important, because they include use cases and details that may not occur to
a developer who just picks it up. It is a great way to contribute!
From: Ryan Hendrickson <ry...@gmail.com>
<ry...@gmail.com>
Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
Date: August 6, 2021 at 03:23:43
To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
If I put Jira's in, then I'd have to admit I'm a developer vs a user....
:-)
I'm just a power user at heart :-) ...
Ryan
On Thu, Aug 5, 2021 at 2:01 PM Otto Fowler <ot...@gmail.com> wrote:
> Ryan, these are awesome, are there jiras for them? It would be best to
> capture the requirements / use cases
>
> From: Ryan Hendrickson <ry...@gmail.com>
> <ry...@gmail.com>
> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> Date: August 4, 2021 at 01:14:23
> To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
>
> I'll preface with what I'm about to say is not rooted in any real
> understanding of the lift required or other customer experiences... We
> have used NiFi for a long time and are excited for what new features may
> lay ahead in the 2.x line.
>
> ---After having written this, I think our succinct pain points focus
> around large data flow management, auto-scaling container environments,
and
> integration of external analytic/enrichment creation.---
>
> Thanks,
> Ryan
>
>
> 1 - Integration with auto-scaling container environments
> We often struggle with scaling custom processors in large flows. We've
> implemented a recent design pattern where we write custom python code
> packaged into a docker image, fronted with a web-service, and deployed
into
> a container environment. In NiFi we use InvokeHTTP to call out to the
> web-service. If the load becomes too great, we can easily spin-up more
> than one external web-service without sacrificing the NiFi's overall
> thruput and thread contention. In this scenario, we're using NiFi to
> manage the overall dataflow, routes, ETL services, etc, but not so much
> smarts around auto-scaling processing, etc.
>
> This makes it really easy to receive a dataflow analytic/enrichment from
an
> external team, wrap it in a docker image, and deploy it into an
> auto-scaling container environment. This also lets us avoid conversations
> about how performant the code is and if it's meeting a minimum throughput
> requirement - typically XXGB/5minutes.
>
> The request for 2.x would be to make interacting with container deployed
> services a more seamless, streamlined process, possibly through a
> normalized API or as a "Remote Container Environment" that can bind to a
> service with a health endpoint and verify it's alive and well, etc.
>
> 2 - Integration of S3 buckets with Remote Process Groups for outage
events
> and data overruns
> Another frequent struggle is the occasional death of a server in a NiFi
> flow. Example -- Our overall NiFi flow consists of ~15 stand-alone NiFis
> and 2 NiFi clusters. The stand-alone NiFi's are linearly chained together
> with Remote Process Groups to perform the overall dataflow goals. If any
> one of the 15 experience an outage, data begins to backup on the previous
> NiFi, hitting backpressure limits, etc. When the down server comes back
> online, it receives a flood of data, often overwhelming it, necessitating
> careful flowfile limits and backpressure points.
>
> The design pattern we're exploring, but would like to see baked into
NiFi,
> is that when a Remote Process Group becomes unavailable due to
> server-to-server comms failures AND the backpressure limit is reached on
a
> relationship, the Remote Process Group would begin sending data to an S3
> bucket as an automatic failover procedure. When the downed server comes
> back-online, the Remote Process Group port would know to pull data from
S3
> AND receive data via the port from the chained server.
>
> 3 - Large Canvas Management
> Beginning with a NiFi flow that consists of ~15 stand-alone NiFis and 2
> NiFi clusters, each canvas is managed individually. To see the entire
> canvas, systems engineering diagrams need to be kept up-to-date
identifying
> all the input/output ports. To move processors from server to server
often
> requires the verification of the existence of scripts, configuration
files,
> NiFi custom nars, and using the templating or Registry to migrate
portions
> of the dataflow.
>
> Our current design pattern to accomplish some of these tasks include
using
> Ansible, Jenkins, and Nexus to verify script & configuration files are up
> to date. We use Confluence's Gliffy to document the NiFi communication
> paths between the servers.
>
> Ideally, we'd be able to see an overall management view of the Canvases
> together in a single place - it would indicate which server the portions
of
> the canvas are deployed on and tie together where the Remote Process
Group
> ports are connected. Additionally, NiFi would allow the user to move
> Processors and Process Groups seamlessly from one server to another
server
> to help manually distribute and balance data flow processing cycles.
>
>
>
> On Tue, Aug 3, 2021 at 12:45 PM Joe Witt <jo...@gmail.com> wrote:
>
> > Kevin,
> >
> > This may well be a very interesting approach which opens up a lot of
> > options for us in how we tackle a 2.x...
> >
> > Thanks
> >
> > On Tue, Aug 3, 2021 at 8:45 AM Kevin Doran <kd...@gmail.com>
> > wrote:
> > >
> > > Outside of core NiFi, for MiNiFi, I think it would be beneficial to
> > align MiNiFi Java and MiNiFI C++ to use the same method(s) for command
> and
> > control / flow deployment. This was discussed when MiNiFi Java was
merged
> > into the NiFi codebase [1].
> > >
> > > My proposal would be to create a Java implementation of the C2
Protocol
> > used by MiNiFi C++, both server-side and client-side, to allow
deploying
> to
> > MiNiFi Java instances from a centralize flow definition. Thanks to the
> > unification of MiNiFi Java and NiFi, this could likely also be added as
a
> > capability of NiFi at that same time.
> > >
> > > If we are able to do this on the 1.x line, then I would also support
> > deprecating other methods of flow deployment to MiNiFi outlined in [1]
> and
> > removing them in a 2.x release.
> > >
> > > [1] https://github.com/apache/nifi/pull/4933#issuecomment-808411977 <
> > https://github.com/apache/nifi/pull/4933#issuecomment-808411977>
> > > [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design <
> > https://cwiki.apache.org/confluence/display/MINIFI/C2+Design>
> > >
> > > Thanks,
> > > Kevin
> > >
> > > > On Aug 3, 2021, at 10:07 AM, David Handermann <
> > exceptionfactory@gmail.com> wrote:
> > > >
> > > > Thanks for the continued discussion around what can or should be
> > removed.
> > > > Having a clear migration path from version 1 to version 2 will be
> very
> > > > important for supporting current deployments.
> > > >
> > > > Following the example of some other projects, one way to consider
the
> > > > upgrade is from the point of view of deprecation warnings. If
> > implemented
> > > > correctly, an installation of NiFi running the latest minor release
> of
> > > > version 1, and showing no deprecation warnings, should be
upgradeable
> > to
> > > > version 2 without any changes. Making this work involves accurately
> > tagging
> > > > components and methods with deprecation indicators, and providing a
> > clear
> > > > way to observe deprecation warnings. With the current version
> > containing
> > > > both deprecated components and deprecated methods in various
classes,
> > this
> > > > would involve some work to get right. A simple approach might be a
> > named
> > > > logger for deprecation warnings, which would write a separate log
> > file. A
> > > > more advanced approach might involve a tool to analyze the system
and
> > flow
> > > > configurations. Either way, it seems that some additional work in a
> > minor
> > > > release version is necessary before considering a major release
> > version.
> > > >
> > > > One other point on compatibility: as long as the core component API
> > > > definition remains backwards compatible, it should be possible to
run
> > older
> > > > components. This provides a transition option for customers using
> > > > components such as PostHTTP, or any others that are not actively
> > > > maintained. At some point it may become necessary to break
> > compatibility at
> > > > that level, but at least for version 2, maintaining component API
> > > > compatibility is key.
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Sun, Aug 1, 2021 at 10:23 AM Mark Bean <ma...@gmail.com>
> > wrote:
> > > >
> > > >> I created a JIRA ticket to investigate and improve the performance
> of
> > > >> InvokeHTTP. It includes a flow definition for benchmarking the
> > performance
> > > >> of both PostHTTP and InvokeHTTP.
> > > >>
> > > >> https://issues.apache.org/jira/browse/NIFI-8968
> > > >>
> > > >> Thanks,
> > > >> Mark
> > > >>
> > > >> On Fri, Jul 30, 2021 at 6:41 PM Adam Taft <ad...@adamtaft.com>
> wrote:
> > > >>
> > > >>> I'm not seeing the side thread that was going to discuss
> deprecation
> > of
> > > >>> PostHTTP. Has that thread started and I just don't see it?
> > > >>>
> > > >>> One (significant?) concern with PostHTTP is the smooth
integration
> of
> > > >>> NiFi-to-NiFi communication that is very transparently enabled
with
> > the
> > > >>> ListenHTTP and PostHTTP processors. There's some special logic in
> > there
> > > >> for
> > > >>> handling flowfiles that InvokeHTTP doesn't really (nor should
> really)
> > > >> have.
> > > >>>
> > > >>> I know of several (large) NiFi installations that rely on the
> > PostHTTP /
> > > >>> ListenHTTP combination. It has enabled NiFi to NiFi communication
> for
> > > >> folks
> > > >>> reluctant or unable to enable site-to-site type configuration.
> > > >>>
> > > >>> Honestly, I don't know why we'd want to "deprecate" any
processor,
> as
> > > >>> opposed to just moving it to a new location. If these processors
> can
> > be
> > > >>> ported and maintained to whatever the 2.0 API looks like, there's
> > > >> possibly
> > > >>> little harm keeping them around.
> > > >>>
> > > >>> And by the way, what happened to the "marketplace" concept? Is
this
> > being
> > > >>> considered for 2.0 as well? Because relocating the deprecated
> > processors
> > > >>> to an external nar might be the best solution. Losing PostHTTP
> > entirely I
> > > >>> think would be a mistake, but I'd conceptually support relocating
> it.
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> /Adam
> > > >>>
> > > >>> On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <jo...@gmail.com>
> wrote:
> > > >>>
> > > >>>> Looks like we just need to knock out 5 JIRAs :) [1]
> > > >>>>
> > > >>>> I felt like we had a label folks were using at one point but
> quickly
> > > >>>> looking revealed nothing exciting. After this confluence page
> > > >>>> stabilizes a bit we can probably knock out some JIRAs and such.
> > > >>>>
> > > >>>> [1]
> https://issues.apache.org/jira/projects/NIFI/versions/12339599
> > > >>>>
> > > >>>> On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <
> > ottobackwards@gmail.com>
> > > >>>> wrote:
> > > >>>>>
> > > >>>>> I find myself wishing I had a list of all the jiras / issues
that
> > > >> have
> > > >>>>> been put off for a 2.0 release because they required some
change
> or
> > > >>>> another
> > > >>>>> :(
> > > >>>>>
> > > >>>>> From: Joe Witt <jo...@gmail.com> <jo...@gmail.com>
> > > >>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > >> dev@nifi.apache.org>
> > > >>>>> Date: July 27, 2021 at 12:30:35
> > > >>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
> dev@nifi.apache.org
> > >
> > > >>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > >>>>>
> > > >>>>> A few thoughts:
> > > >>>>>
> > > >>>>> 1. I would love to see deprecation notices show up in the UI in
> > > >>>>> various ways to help motivate users to move off things to more
> > > >>>>> supportable things. That is not a prerequisite for anything
> > happening
> > > >>>>> however. Just a good feature/nice thing to do for users when
> > someone
> > > >>>>> is able to tackle it.
> > > >>>>>
> > > >>>>> 2. The decision to deprecate something and to further remove it
> > need
> > > >>>>> not mean there is a superior solution available. If that thing
> > itself
> > > >>>>> isn't getting the love/attention it needs to be
> > > >>>>> maintained/supported/healthy going forward that alone is enough
> to
> > > >>>>> remove it. That might well be the case with PostHTTP [1] and
for
> > > >>>>> comparison you can see how much effort has gone into InvokeHTTP
> > [2].
> > > >>>>>
> > > >>>>> 3. When discussing a 2.0 release each thing we add as a 'must
do'
> > the
> > > >>>>> further away from reality such a release will become. We'll
have
> to
> > > >>>>> get very specific about 'musts' vs 'wants'.
> > > >>>>>
> > > >>>>> [1]
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> >
>
>
https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
> > > >>>>> [2]
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> >
>
>
https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
> > > >>>>>
> > > >>>>> On Tue, Jul 27, 2021 at 8:47 AM David Handermann
> > > >>>>> <ex...@apache.org> wrote:
> > > >>>>>>
> > > >>>>>> Thanks Mark, providing a template or comparison statistics
with
> > > >> Java
> > > >>>>>> versions and component configuration details would be very
> > helpful.
> > > >>> If
> > > >>>> it
> > > >>>>>> is possible to run tests using a public API or deployable
> service,
> > > >>> that
> > > >>>>>> would also help confirm potential differences.
> > > >>>>>>
> > > >>>>>> Showing a deprecation notice in the UI could be helpful,
perhaps
> > > >> as a
> > > >>>>>> configurable option. NIFI-8650 describes a general Flow
Analysis
> > > >>>>>> capability, and it seems like that might be one possible way
to
> > > >>> surface
> > > >>>>>> deprecation warnings. For something more specific to component
> > > >>>>> deprecation,
> > > >>>>>> it seems important to find a balance between making it obvious
> and
> > > >>>> making
> > > >>>>>> it something that ends up getting ignored.
> > > >>>>>>
> > > >>>>>> Regards,
> > > >>>>>> David Handermann
> > > >>>>>>
> > > >>>>>> On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <
> mark.o.bean@gmail.com
> > >
> > > >>>> wrote:
> > > >>>>>>
> > > >>>>>>> I'll start a new thread for PostHTTP when I get a template
> and/or
> > > >>>>> detailed
> > > >>>>>>> stats.
> > > >>>>>>>
> > > >>>>>>> I know the deprecation is noted in the documentation. That's
a
> > > >>>>> necessary
> > > >>>>>>> and minimum level of notification. I was suggesting it be
more
> > > >>>> obvious
> > > >>>>> in
> > > >>>>>>> the UI. I think it would be beneficial to somehow be aware of
> the
> > > >>>>>>> deprecation status simply by looking at the flow (perhaps
even
> on
> > > >>> the
> > > >>>>>>> summary pages too), and without having to open the
> documentation
> > > >>> for
> > > >>>>> every
> > > >>>>>>> processor to confirm whether or not a component is marked as
> > > >>>>> deprecated.
> > > >>>>>>>
> > > >>>>>>> Thanks,
> > > >>>>>>> Mark
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
> > > >>>>>>> exceptionfactory@apache.org> wrote:
> > > >>>>>>>
> > > >>>>>>>> Mark,
> > > >>>>>>>>
> > > >>>>>>>> Thanks for the feedback. It may be better to start a
separate
> > > >>>> thread
> > > >>>>> on
> > > >>>>>>>> PostHTTP, but can you provide an example flow demonstrating
> the
> > > >>>>>>> performance
> > > >>>>>>>> differences between PostHTTP and InvokeHTTP?
> > > >>>>>>>>
> > > >>>>>>>> PostHTTP uses the Apache HttpComponents library, whereas
> > > >>> InvokeHTTP
> > > >>>>> uses
> > > >>>>>>>> OkHttp. NiFi 1.13.2 and 1.14.0 included major version
updates
> > > >> for
> > > >>>>> OkHttp,
> > > >>>>>>>> so it would be important to test with the most recent
version.
> > > >> It
> > > >>>> is
> > > >>>>> also
> > > >>>>>>>> worth noting that test classes for PostHTTP are now
> integration
> > > >>>> tests
> > > >>>>> as
> > > >>>>>>>> opposed to unit tests, which means they are not executed as
> > > >> part
> > > >>> of
> > > >>>>> the
> > > >>>>>>>> automated builds.
> > > >>>>>>>>
> > > >>>>>>>> The NiFi documentation includes the contents of the
> > > >>>> DeprecationNotice
> > > >>>>> for
> > > >>>>>>>> PostHTTP:
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> >
>
>
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
> > > >>>>>>>>
> > > >>>>>>>> Regards,
> > > >>>>>>>> David Handermann
> > > >>>>>>>>
> > > >>>>>>>> On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <
> > > >> mark.o.bean@gmail.com
> > > >>>>
> > > >>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> I'm strongly in favor of reducing tech debt, and moving
> > > >>>>> deliberately
> > > >>>>>>> to a
> > > >>>>>>>>> 2.0 release. I have a concern with just one processor that
is
> > > >>>>> currently
> > > >>>>>>>>> marked as deprecated: PostHTTP. (I have not evaluated
> > > >>>> specifically
> > > >>>>> any
> > > >>>>>>>>> other deprecated components; I cannot say if there are or
are
> > > >>> not
> > > >>>>>>> similar
> > > >>>>>>>>> issues elsewhere.) I understand the rationale for
deprecating
> > > >>>> this
> > > >>>>>>>>> processor in that it eliminates a processor whose
> > > >> functionality
> > > >>>> is
> > > >>>>>>>>> available elsewhere, specifically in the more flexible
> > > >>>> InvokeHTTP.
> > > >>>>>>>> However,
> > > >>>>>>>>> in my experience and testing, PostHTTP performs transfers
> > > >> with
> > > >>>> far
> > > >>>>>>>> greater
> > > >>>>>>>>> throughput than InvokeHTTP. I would not be in favor of
> > > >> removing
> > > >>>>>>> PostHTTP
> > > >>>>>>>>> unless/until InvokeHTTP is refactored to increase its
> > > >>> throughput
> > > >>>>>>>>> capability.
> > > >>>>>>>>>
> > > >>>>>>>>> Has anyone else continued to use PostHTTP over InvokeHTTP
for
> > > >>>>> similar
> > > >>>>>>>>> reasons? Or, is there a performance-related configuration
for
> > > >>>>>>> InvokeHTTP
> > > >>>>>>>> I
> > > >>>>>>>>> may have missed?
> > > >>>>>>>>>
> > > >>>>>>>>> Also, in order to help facilitate a smooth transition to
2.0
> > > >>>> from a
> > > >>>>>>> user
> > > >>>>>>>>> perspective, would it be advisable to add some sort of
visual
> > > >>>>> indicator
> > > >>>>>>>> in
> > > >>>>>>>>> the UI for components that are currently annotated as
> > > >>>> @Deprecated?
> > > >>>>> This
> > > >>>>>>>>> might help users proactively modify their flow prior to a
> > > >>> release
> > > >>>>> that
> > > >>>>>>>>> would otherwise break it.
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <
> > > >> bbende@gmail.com>
> > > >>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> With the merging of MiNiFi and registry into the NiFi
repo,
> > > >>>> we've
> > > >>>>>>>>>> actually gone more towards a mono-repo where everything is
> > > >>>>> released
> > > >>>>>>>>>> together. Eventually it would still be nice to have a
> > > >> smaller
> > > >>>>> base
> > > >>>>>>>>>> distribution containing just the framework and standard
> > > >> NARs,
> > > >>>> but
> > > >>>>> it
> > > >>>>>>>>>> is hard to tackle that until we provide a way for users to
> > > >>>> easily
> > > >>>>>>>>>> obtain the NARs in some other way.
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <
> > > >>>>> edward.armes@gmail.com
> > > >>>>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Given the major version number shift and the spliting up
> > > >> of
> > > >>>>>>>> processors
> > > >>>>>>>>>> into
> > > >>>>>>>>>>> multiple NAR's. I'd like to suggest that we start
> > > >>>> individually
> > > >>>>>>>>> versioning
> > > >>>>>>>>>>> individual NARs/ bundles.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I can see this bringing a large number of benifits
> > > >>> including
> > > >>>>> making
> > > >>>>>>>>> Nifi
> > > >>>>>>>>>>> more deployable with things RPM, but also potentially
> > > >>>> allowing
> > > >>>>>>> those
> > > >>>>>>>>> that
> > > >>>>>>>>>>> have to deploy managed Nifi instances easier to upgrade.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Edward
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Mon, 26 Jul 2021, 20:42 Otto Fowler, <
> > > >>>> ottobackwards@gmail.com>
> > > >>>>>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> The issue with updating the aws sdk is if it breaks any
> > > >>> one
> > > >>>>> of
> > > >>>>>>> the
> > > >>>>>>>>>>>> processors.
> > > >>>>>>>>>>>> the Web Gateway API invoke processor for example is not
> > > >>>> using
> > > >>>>> a
> > > >>>>>>>> high
> > > >>>>>>>>>> level
> > > >>>>>>>>>>>> purpose build client and may break.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> If we change the aws version, we need to coordinate in
> > > >>>> such a
> > > >>>>> way
> > > >>>>>>>>> that
> > > >>>>>>>>>> they
> > > >>>>>>>>>>>> all
> > > >>>>>>>>>>>> can come along reasonably.
> > > >>>>>>>>>>>> IE: what happens if 1 or 2 break but the rest or OK?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> From: David Handermann <ex...@apache.org>
> > > >>>>>>>>>>>> <ex...@apache.org>
> > > >>>>>>>>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > >>>>>>>>> dev@nifi.apache.org>
> > > >>>>>>>>>>>> Date: July 26, 2021 at 09:33:42
> > > >>>>>>>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > >>>>>>> dev@nifi.apache.org
> > > >>>>>>>>>
> > > >>>>>>>>>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Chris,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks for the reply and recommendations. It seems like
> > > >>>> some
> > > >>>>> of
> > > >>>>>>> the
> > > >>>>>>>>>> work to
> > > >>>>>>>>>>>> reorganize the module structure could be done outside
> > > >> of
> > > >>> a
> > > >>>>> major
> > > >>>>>>>>>> release,
> > > >>>>>>>>>>>> but it would be great to target any breaking changes
> > > >> for
> > > >>>> 2.0.
> > > >>>>>>>>> Perhaps a
> > > >>>>>>>>>>>> separate feature proposal on module restructuring, with
> > > >>> the
> > > >>>>> goal
> > > >>>>>>> of
> > > >>>>>>>>>>>> supporting optimized builds, would be a helpful way to
> > > >>> move
> > > >>>>> that
> > > >>>>>>>> part
> > > >>>>>>>>>> of
> > > >>>>>>>>>>>> the discussion forward.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Regarding updating AWS SDK to version 2, it seems like
> > > >>> that
> > > >>>>> might
> > > >>>>>>>> be
> > > >>>>>>>>>>>> possible now. I haven't taken a close look at the
> > > >>>> referencing
> > > >>>>>>>>>> components,
> > > >>>>>>>>>>>> so I'm not sure about the level of effort involved.
> > > >> Minor
> > > >>>>> NiFi
> > > >>>>>>>>> version
> > > >>>>>>>>>>>> updates have incorporated major new versions of
> > > >>>> dependencies.
> > > >>>>> For
> > > >>>>>>>>>> example,
> > > >>>>>>>>>>>> NiFi 1.14 included an upgrade from Spring Framework 4
> > > >> to
> > > >>> 5.
> > > >>>>> On
> > > >>>>>>> the
> > > >>>>>>>>> one
> > > >>>>>>>>>>>> hand, including the AWS SDK update as part of a major
> > > >>>> release
> > > >>>>>>> seems
> > > >>>>>>>>>>>> helpful, but unless there are changes that break
> > > >> existing
> > > >>>>>>> component
> > > >>>>>>>>>>>> properties, upgrading the AWS SDK could be worked
> > > >>>>> independently.
> > > >>>>>>>>>> Others may
> > > >>>>>>>>>>>> have more insight into particular usage of that
> > > >> library.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>> David Handermann
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > > >>>>>>>>>>>> <ch...@naimuri.com.invalid> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Might be worth considering refactoring the build as
> > > >>> part
> > > >>>> of
> > > >>>>>>> this
> > > >>>>>>>>> work
> > > >>>>>>>>>>>> too,
> > > >>>>>>>>>>>>> e.g. only building the bits of the repo affected by a
> > > >>>>> commit,
> > > >>>>>>>> etc.
> > > >>>>>>>>> -
> > > >>>>>>>>>>>>> discussed briefly in previous threads but don't think
> > > >>> any
> > > >>>>>>> changes
> > > >>>>>>>>>> made
> > > >>>>>>>>>>>> yet.
> > > >>>>>>>>>>>>> If NARs/components are likely to be split up and
> > > >>>> refactored
> > > >>>>>>> then
> > > >>>>>>>>> such
> > > >>>>>>>>>>>> work
> > > >>>>>>>>>>>>> around the build probably makes sense to consider.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I've a couple of PRs open that include updates to
> > > >>>>> Elasticsearch
> > > >>>>>>>>>> versions
> > > >>>>>>>>>>>>> already, although I stopped at 7.10.2 (after which
> > > >>>> Elastic
> > > >>>>>>>> changed
> > > >>>>>>>>>>>> licence)
> > > >>>>>>>>>>>>> in case there were licence concerns. But more work
> > > >> can
> > > >>> be
> > > >>>>> done
> > > >>>>>>> to
> > > >>>>>>>>>> tidy up
> > > >>>>>>>>>>>>> the processors, absolutely.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> AWS libraries to v2 would seem a sensible move and a
> > > >>>>> refactor
> > > >>>>>>> of
> > > >>>>>>>>>> those
> > > >>>>>>>>>>>>> processors as well.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Chris Sampson
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > > >>>>>>>>>>>> exceptionfactory@apache.org>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks for pointing out the standard NAR bundles,
> > > >>> Mark.
> > > >>>>> There
> > > >>>>>>>>> are a
> > > >>>>>>>>>>>>> number
> > > >>>>>>>>>>>>>> of components in the standard NAR bundles with
> > > >>>> particular
> > > >>>>>>>>>> dependencies
> > > >>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>> would make more sense in separate NARs.
> > > >> Reorganizing
> > > >>>> the
> > > >>>>>>>> standard
> > > >>>>>>>>>> NAR
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>> components with limited dependencies and wide
> > > >>>>> applicability
> > > >>>>>>>> would
> > > >>>>>>>>>>>>>> definitely help with future maintenance.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>>>> David Handermann
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> > > >>>>>>>>> markap14@hotmail.com>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> There’s also some code that exists in order to
> > > >>>> maintain
> > > >>>>>>>>> backward
> > > >>>>>>>>>>>>>>> compatibility in the repositories. I would very
> > > >>> much
> > > >>>>> like
> > > >>>>>>> the
> > > >>>>>>>>>>>>>> repositories
> > > >>>>>>>>>>>>>>> to contain no unnecessary code. And swap file
> > > >>> format
> > > >>>>>>> supports
> > > >>>>>>>>>> really
> > > >>>>>>>>>>>>> old
> > > >>>>>>>>>>>>>>> formats. And the old impls of the repositories
> > > >>>>> themselves,
> > > >>>>>>>> like
> > > >>>>>>>>>>>>>>> PersistentProvRepo instead of WriteAheadProv
> > > >> Repo,
> > > >>>> etc.
> > > >>>>>>> Lots
> > > >>>>>>>> of
> > > >>>>>>>>>> stuff
> > > >>>>>>>>>>>>>> there
> > > >>>>>>>>>>>>>>> that can be removed. And some methods in
> > > >>>> ProcessSession
> > > >>>>>>> that
> > > >>>>>>>>> are
> > > >>>>>>>>>>>> never
> > > >>>>>>>>>>>>>> used
> > > >>>>>>>>>>>>>>> by any processor in the codebase but exists in
> > > >> the
> > > >>>>> public
> > > >>>>>>> API
> > > >>>>>>>>> so
> > > >>>>>>>>>>>> can’t
> > > >>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>> removed till 2.0.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I think his is also a great time to clean up the
> > > >>>>> “standard
> > > >>>>>>>>> nar.”
> > > >>>>>>>>>> At
> > > >>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>> point, it’s something like 70 MB. And many of the
> > > >>>>>>> components
> > > >>>>>>>>>> there
> > > >>>>>>>>>>>> are
> > > >>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>> really “standard” - things like connecting to
> > > >> FTP &
> > > >>>>> SFTP
> > > >>>>>>>>>> servers, XML
> > > >>>>>>>>>>>>>>> processing, Jolt transform, etc. could
> > > >> potentially
> > > >>> be
> > > >>>>> moved
> > > >>>>>>>>> into
> > > >>>>>>>>>>>> other
> > > >>>>>>>>>>>>>>> nars. The
> > > >>>>> nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
> > > >>>>>>> is
> > > >>>>>>>>>> 6.9 MB
> > > >>>>>>>>>>>> is
> > > >>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>> necessary for stateless or minifi java. Lots of
> > > >>>> things
> > > >>>>>>>> probably
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>>>>> reconsider within the standard nar.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I definitely think this is a reasonable approach,
> > > >>> to
> > > >>>>> allow
> > > >>>>>>>> for
> > > >>>>>>>>> a
> > > >>>>>>>>>> 2.0
> > > >>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>> is not a huge feature release but allows the
> > > >>> project
> > > >>>> to
> > > >>>>> be
> > > >>>>>>>>>> simpler
> > > >>>>>>>>>>>> and
> > > >>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>> nimble.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks
> > > >>>>>>>>>>>>>>> -Mark
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > > >>>>>>>>>> mikerthomsen@gmail.com
> > > >>>>>>>>>>>>>> <mailto:
> > > >>>>>>>>>>>>>>> mikerthomsen@gmail.com>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Russell,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> AFAICT from looking at Elastic's repos, the low
> > > >>> level
> > > >>>>> REST
> > > >>>>>>>>>> client is
> > > >>>>>>>>>>>>>>> still fine.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> >
>
>
https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Our Elasticsearch support is spread over two NARs
> > > >>> at
> > > >>>>>>> present.
> > > >>>>>>>>> One
> > > >>>>>>>>>>>> uses
> > > >>>>>>>>>>>>>>> OkHttp and the other uses that low level Elastic
> > > >>> REST
> > > >>>>>>> client.
> > > >>>>>>>>>>>>>>> Therefore, I think we're fine on licensing for
> > > >> the
> > > >>>>> moment.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Mike
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > > >>>>>>>>>>>> russ@windofkeltia.com
> > > >>>>>>>>>>>>>>> <ma...@windofkeltia.com>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Bringing up Elastic also reminds me that the
> > > >>> Elastic
> > > >>>>>>>> framework
> > > >>>>>>>>>> has
> > > >>>>>>>>>>>> just
> > > >>>>>>>>>>>>>>> recently transitioned out of Open Source, so to
> > > >>>>> acknowledge
> > > >>>>>>>>> that,
> > > >>>>>>>>>>>> maybe
> > > >>>>>>>>>>>>>>> some effort toward OpenSearch--I say this not
> > > >>>>> understanding
> > > >>>>>>>>>> exactly
> > > >>>>>>>>>>>> how
> > > >>>>>>>>>>>>>>> this sort of thing is considered in a
> > > >> large-scale,
> > > >>>>>>>> world-class
> > > >>>>>>>>>>>> software
> > > >>>>>>>>>>>>>>> project like Apache NiFi. (I'm not a contributor,
> > > >>>> just
> > > >>>>> a
> > > >>>>>>>>> grateful
> > > >>>>>>>>>>>>>>> consumer.)
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Russ
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > >>>>>>>>>>>>>>> Along with the itemized list for ancient
> > > >> components
> > > >>>> we
> > > >>>>>>> should
> > > >>>>>>>>>> look at
> > > >>>>>>>>>>>>>>> updating versions of drivers, SDKs, etc. for
> > > >>> external
> > > >>>>>>> systems
> > > >>>>>>>>>> such as
> > > >>>>>>>>>>>>>>> Elasticsearch, Cassandra, etc. There may be
> > > >>> breaking
> > > >>>>>>> changes
> > > >>>>>>>>> but
> > > >>>>>>>>>> 2.0
> > > >>>>>>>>>>>>>>> is probably the right time to get things up to
> > > >> date
> > > >>>> to
> > > >>>>> make
> > > >>>>>>>>> them
> > > >>>>>>>>>> more
> > > >>>>>>>>>>>>>>> useful to more people.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > > >>>>>>>>>> thenatog@gmail.com
> > > >>>>>>>>>>>>>> <mailto:
> > > >>>>>>>>>>>>>>> thenatog@gmail.com>> wrote:
> > > >>>>>>>>>>>>>>> I'm a +1 for removing pretty much all of this
> > > >>> stuff.
> > > >>>>> There
> > > >>>>>>>> are
> > > >>>>>>>>>>>> security
> > > >>>>>>>>>>>>>>> implications to keeping old dependencies around,
> > > >> so
> > > >>>> the
> > > >>>>>>> more
> > > >>>>>>>>> old
> > > >>>>>>>>>> code
> > > >>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>> can remove the better. I agree that eventually we
> > > >>>> need
> > > >>>>> to
> > > >>>>>>>> move
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>>>> supporting only Java 11+, and as our next release
> > > >>>> will
> > > >>>>>>>> probably
> > > >>>>>>>>>> be
> > > >>>>>>>>>>>>> about
> > > >>>>>>>>>>>>>> 4
> > > >>>>>>>>>>>>>>> - 6 months from now that doesn't seem too soon.
> > > >> We
> > > >>>>> could
> > > >>>>>>>>>> potentially
> > > >>>>>>>>>>>>>> break
> > > >>>>>>>>>>>>>>> this in two and remove the deprecated processors
> > > >>> and
> > > >>>>> leave
> > > >>>>>>>> 1.x
> > > >>>>>>>>> on
> > > >>>>>>>>>>>> Java
> > > >>>>>>>>>>>>> 8,
> > > >>>>>>>>>>>>>>> and finally start on 2.x which would support only
> > > >>>> Java
> > > >>>>> 11.
> > > >>>>>>>> I'm
> > > >>>>>>>>>> unsure
> > > >>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>> what implications changing the date and time
> > > >>> handling
> > > >>>>> would
> > > >>>>>>>>> have
> > > >>>>>>>>>> -
> > > >>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>> running systems that use long term historical
> > > >> logs,
> > > >>>>>>>> unexpected
> > > >>>>>>>>>>>> impacts
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> time logging could be a problem.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> As Joe says I think feature work will have to be
> > > >>>>> dedicated
> > > >>>>>>> to
> > > >>>>>>>>>> 2.x and
> > > >>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>> could support 1.x for security fixes for some
> > > >>> period
> > > >>>> of
> > > >>>>>>> time.
> > > >>>>>>>>> 2.x
> > > >>>>>>>>>>>> seems
> > > >>>>>>>>>>>>>>> like a gargantuan task but it's probably time to
> > > >>> get
> > > >>>>>>> started.
> > > >>>>>>>>> Not
> > > >>>>>>>>>>>> sure
> > > >>>>>>>>>>>>>> how
> > > >>>>>>>>>>>>>>> we handle all open PRs and the transition between
> > > >>> 1.x
> > > >>>>> and
> > > >>>>>>>> 2.x.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> > > >>>>>>>> joe.witt@gmail.com
> > > >>>>>>>>>>>> <mailto:
> > > >>>>>>>>>>>>>>> joe.witt@gmail.com>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Jon
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> You're right we have to be careful and you're
> > > >> right
> > > >>>>> there
> > > >>>>>>> are
> > > >>>>>>>>>> still
> > > >>>>>>>>>>>>>>> significant Java 8 users out there. But we also
> > > >>> have
> > > >>>> to
> > > >>>>> be
> > > >>>>>>>>>> careful
> > > >>>>>>>>>>>>>>> about security and sustainability of the
> > > >> codebase.
> > > >>> If
> > > >>>>> we
> > > >>>>>>> had
> > > >>>>>>>>>> talked
> > > >>>>>>>>>>>>>>> about this last year when that article came out
> > > >> I'd
> > > >>>>> have
> > > >>>>>>>> agreed
> > > >>>>>>>>>> it is
> > > >>>>>>>>>>>>>>> too early. Interestingly that link seems to get
> > > >>>> updated
> > > >>>>>>> and I
> > > >>>>>>>>>> tried
> > > >>>>>>>>>>>>>>> [1] and found more recent data (not sure how
> > > >>> recent).
> > > >>>>>>> Anyway
> > > >>>>>>>> it
> > > >>>>>>>>>>>>>>> suggests Java 8 is still the top dog but we see
> > > >>> good
> > > >>>>> growth
> > > >>>>>>>> on
> > > >>>>>>>>>> 11. In
> > > >>>>>>>>>>>>>>> my $dayjob this aligns to what I'm seeing too.
> > > >>>>> Customers
> > > >>>>>>>> didn't
> > > >>>>>>>>>> seem
> > > >>>>>>>>>>>>>>> to care about Java 11 until later half last year
> > > >>> and
> > > >>>>> now
> > > >>>>>>>>>> suddenly it
> > > >>>>>>>>>>>>>>> is all over the place.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I think once we put out a NiFi 2.0 release we'd
> > > >> see
> > > >>>>> rapid
> > > >>>>>>>>>> decrease in
> > > >>>>>>>>>>>>>>> work on the 1.x line just being blunt. We did
> > > >> this
> > > >>>> many
> > > >>>>>>> years
> > > >>>>>>>>> ago
> > > >>>>>>>>>>>>>>> with 0.x to 1.x and we stood behind 0.x for a
> > > >> while
> > > >>>>> (maybe
> > > >>>>>>> a
> > > >>>>>>>>>> year or
> > > >>>>>>>>>>>>>>> so) but it was purely bug fix/security related
> > > >>> bits.
> > > >>>> We
> > > >>>>>>> would
> > > >>>>>>>>>> need to
> > > >>>>>>>>>>>>>>> do something similar. But feature work would
> > > >> almost
> > > >>>>>>> certainly
> > > >>>>>>>>> go
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>>>>> the 2.x line. Maybe there are other workable
> > > >> models
> > > >>>> but
> > > >>>>> my
> > > >>>>>>>>>> instinct
> > > >>>>>>>>>>>>>>> suggests this is likely to follow a similar path.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> ...anyway I agree it isn't that easy of a call to
> > > >>>> dump
> > > >>>>> Java
> > > >>>>>>>> 8.
> > > >>>>>>>>> We
> > > >>>>>>>>>>>>>>> need to make the call in both the interests of
> > > >> the
> > > >>>> user
> > > >>>>>>> base
> > > >>>>>>>>> and
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>>> contributor base of the community.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> [1]
> > > >>>> https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks
> > > >>>>>>>>>>>>>>> Joe
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
> > > >>>>>>> joe.witt@gmail.com
> > > >>>>>>>>>> <mailto:
> > > >>>>>>>>>>>>>>> joe.witt@gmail.com>> wrote:
> > > >>>>>>>>>>>>>>> Russ
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Yeah the flow registry is a key part of it. But
> > > >>> also
> > > >>>>> now
> > > >>>>>>> you
> > > >>>>>>>>> can
> > > >>>>>>>>>>>>>>> download the flow definition in JSON (upload i
> > > >>> think
> > > >>>> is
> > > >>>>>>> there
> > > >>>>>>>>> now
> > > >>>>>>>>>>>>>>> too). Templates offered a series of challenges
> > > >> such
> > > >>>> as
> > > >>>>> we
> > > >>>>>>>> store
> > > >>>>>>>>>> them
> > > >>>>>>>>>>>>>>> in the flow definition which has made flows
> > > >> massive
> > > >>>> in
> > > >>>>> an
> > > >>>>>>>>>> unintended
> > > >>>>>>>>>>>>>>> way which isn't fun for cluster behavior.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> We have a couple cases where we headed down a
> > > >>>>> particular
> > > >>>>>>>>> concept
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>>>>> came up with better approaches later. We need to
> > > >>>>> reconcile
> > > >>>>>>>>> these
> > > >>>>>>>>>> with
> > > >>>>>>>>>>>>>>> the benefit of hindsight, and while being careful
> > > >>> to
> > > >>>> be
> > > >>>>> not
> > > >>>>>>>>>> overly
> > > >>>>>>>>>>>>>>> disruptive to existing users, to reduce the
> > > >>>>>>>>> codebase/maintenance
> > > >>>>>>>>>>>>>>> burden and allow continued evolution of the
> > > >>> project.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > > >>>>>>>>>>>> russ@windofkeltia.com
> > > >>>>>>>>>>>>>>> <ma...@windofkeltia.com>>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>> Joe,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I apologize for the off-topic intrusion, but what
> > > >>>>> replaces
> > > >>>>>>>>>> templates?
> > > >>>>>>>>>>>>>>> The Registry? Templates rocked and we have used
> > > >>> them
> > > >>>>> since
> > > >>>>>>>>> 0.5.x.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Russ
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On 7/23/21 8:31 AM, Joe Witt wrote:
> > > >>>>>>>>>>>>>>> David,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I think this is a highly reasonable approach and
> > > >>>> such a
> > > >>>>>>> focus
> > > >>>>>>>>>> will
> > > >>>>>>>>>>>>>>> greatly help make a 2.0 release far more
> > > >>> approachable
> > > >>>>> to
> > > >>>>>>>> knock
> > > >>>>>>>>>> out.
> > > >>>>>>>>>>>>>>> Not only that but tech debt reduction would help
> > > >>> make
> > > >>>>> work
> > > >>>>>>>>>> towards
> > > >>>>>>>>>>>>>>> major features we'd think about in a 'major
> > > >>> release'
> > > >>>>> sense
> > > >>>>>>>> more
> > > >>>>>>>>>>>>>>> approachable.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> We should remove all deprecated things (as well
> > > >> as
> > > >>>>> verify
> > > >>>>>>> we
> > > >>>>>>>>>> have the
> > > >>>>>>>>>>>>>>> right list). We should remove/consider removal of
> > > >>>>>>> deprecated
> > > >>>>>>>>>>>>>>> concepts
> > > >>>>>>>>>>>>>>> like templates. We should consider whether we can
> > > >>>>> resolve
> > > >>>>>>> the
> > > >>>>>>>>>>>>>>> various
> > > >>>>>>>>>>>>>>> ways we've handled what are now parameters down
> > > >> to
> > > >>>> one
> > > >>>>>>> clean
> > > >>>>>>>>>>>>>>> approach.
> > > >>>>>>>>>>>>>>> We should remove options in the nifi.properties
> > > >>> which
> > > >>>>> turn
> > > >>>>>>>> out
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>>>> never be used quite right (if there are). There
> > > >> is
> > > >>>>> quite a
> > > >>>>>>>> bit
> > > >>>>>>>>> we
> > > >>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>> do purely in the name of tech debt reduction.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Lots to consider here but I think this is the
> > > >> right
> > > >>>>>>>> discussion.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Than ks
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> > > >>>>>>>> bbende@gmail.com
> > > >>>>>>>>>>>> <mailto:
> > > >>>>>>>>>>>>>>> bbende@gmail.com>>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>> I'm a +1 for this... Not sure if this falls under
> > > >>>>> "Removing
> > > >>>>>>>>>>>>>>> Deprecated
> > > >>>>>>>>>>>>>>> Components", but I think we should also look at
> > > >>>>> anything
> > > >>>>>>> that
> > > >>>>>>>>> has
> > > >>>>>>>>>>>>>>> been
> > > >>>>>>>>>>>>>>> marked as deprecated throughout the code base as
> > > >> a
> > > >>>>>>> candidate
> > > >>>>>>>>> for
> > > >>>>>>>>>>>>>>> removal. There are quite a few classes, methods,
> > > >>>>>>> properties,
> > > >>>>>>>>> etc
> > > >>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>> have been waiting for a chance to be removed.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > >>>>>>>>>>>>>>> <exceptionfactory@apache.org<mailto:
> > > >>>>>>>>> exceptionfactory@apache.org
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>> Team,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> With all of the excellent work that many have
> > > >>>>> contributed
> > > >>>>>>> to
> > > >>>>>>>>> NiFi
> > > >>>>>>>>>>>>>>> over the
> > > >>>>>>>>>>>>>>> years, the code base has also accumulated some
> > > >>> amount
> > > >>>>> of
> > > >>>>>>>>>> technical
> > > >>>>>>>>>>>>>>> debt. A
> > > >>>>>>>>>>>>>>> handful of components have been marked as
> > > >>> deprecated,
> > > >>>>> and
> > > >>>>>>>> some
> > > >>>>>>>>>>>>>>> components
> > > >>>>>>>>>>>>>>> remain in the code base to support integration
> > > >> with
> > > >>>> old
> > > >>>>>>>>> versions
> > > >>>>>>>>>>>>>>> of various
> > > >>>>>>>>>>>>>>> products. Following the principles of semantic
> > > >>>>> versioning,
> > > >>>>>>>>>>>>>>> introducing a
> > > >>>>>>>>>>>>>>> major release would provide the opportunity to
> > > >>> remove
> > > >>>>> these
> > > >>>>>>>>>>>>>>> deprecated and
> > > >>>>>>>>>>>>>>> unsupported components.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Rather than focusing the next major release on
> > > >> new
> > > >>>>>>> features,
> > > >>>>>>>>> what
> > > >>>>>>>>>>>>>>> do you
> > > >>>>>>>>>>>>>>> think about focusing on technical debt removal?
> > > >>> This
> > > >>>>>>> approach
> > > >>>>>>>>>>>>>>> would not
> > > >>>>>>>>>>>>>>> make for the most interesting release, but it
> > > >>>> provides
> > > >>>>> the
> > > >>>>>>>>>>>>>>> opportunity to
> > > >>>>>>>>>>>>>>> clean up elements that involve breaking changes.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Focusing on technical debt, at least three
> > > >> primary
> > > >>>>> goals
> > > >>>>>>> come
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>>>> mind for
> > > >>>>>>>>>>>>>>> the next major release:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> 1. Removal of deprecated and unmaintained
> > > >>> components
> > > >>>>>>>>>>>>>>> 2. Require Java 11 as the minimum supported
> > > >> version
> > > >>>>>>>>>>>>>>> 3. Transition internal date and time handling to
> > > >>> JSR
> > > >>>>> 310
> > > >>>>>>>>>> java.time
> > > >>>>>>>>>>>>>>> components
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> *Removing Deprecated Components*
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Removing support for older and deprecated
> > > >>> components
> > > >>>>>>>> provides a
> > > >>>>>>>>>>>>>>> great
> > > >>>>>>>>>>>>>>> opportunity to improve the overall security
> > > >> posture
> > > >>>>> when it
> > > >>>>>>>>> comes
> > > >>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> maintaining dependencies. The OWASP dependency
> > > >>> plugin
> > > >>>>>>> report
> > > >>>>>>>>>>>>>>> currently
> > > >>>>>>>>>>>>>>> generates 50 MB of HTML for questionable
> > > >>>> dependencies,
> > > >>>>> many
> > > >>>>>>>> of
> > > >>>>>>>>>>>>>>> which are
> > > >>>>>>>>>>>>>>> related to old versions of various libraries.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> As a starting point, here are a handful of
> > > >>> components
> > > >>>>> and
> > > >>>>>>>>>>>>>>> extension modules
> > > >>>>>>>>>>>>>>> that could be targeted for removal in a major
> > > >>>> version:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> - PostHTTP and GetHTTP
> > > >>>>>>>>>>>>>>> - ListenLumberjack and the entire
> > > >>>>> nifi-lumberjack-bundle
> > > >>>>>>>>>>>>>>> - ListenBeats and the entire nifi-beats-bundle
> > > >>>>>>>>>>>>>>> - Elasticsearch 5 components
> > > >>>>>>>>>>>>>>> - Hive 1 and 2 components
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> *Requiring Java 11*
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Java 8 is now over seven years old, and NiFi has
> > > >>>>> supported
> > > >>>>>>>>>> general
> > > >>>>>>>>>>>>>>> compatibility with Java 11 for several years.
> > > >> NiFi
> > > >>>>> 1.14.0
> > > >>>>>>>>>>>>>>> incorporated
> > > >>>>>>>>>>>>>>> internal improvements specifically related to TLS
> > > >>>> 1.3,
> > > >>>>>>> which
> > > >>>>>>>>>>>>>>> allowed
> > > >>>>>>>>>>>>>>> closing out the long-running Java 11
> > > >> compatibility
> > > >>>> epic
> > > >>>>>>>>>> NIFI-5174.
> > > >>>>>>>>>>>>>>> Making
> > > >>>>>>>>>>>>>>> Java 11 the minimum required version provides the
> > > >>>>>>> opportunity
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>>>> address
> > > >>>>>>>>>>>>>>> any lingering edge cases and put NiFi in a better
> > > >>>>> position
> > > >>>>>>> to
> > > >>>>>>>>>>>>>>> support
> > > >>>>>>>>>>>>>>> current Java versions.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> *JSR 310 for Date and Time Handling*
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Without making the scope too broad, transitioning
> > > >>>>> internal
> > > >>>>>>>> date
> > > >>>>>>>>>>>>>>> and time
> > > >>>>>>>>>>>>>>> handling to use DateTimeFormatter instead of
> > > >>>>>>> SimpleDateFormat
> > > >>>>>>>>>>>>>>> would provide
> > > >>>>>>>>>>>>>>> a number of advantages. The Java Time components
> > > >>>>> provide
> > > >>>>>>> much
> > > >>>>>>>>>>>>>>> better
> > > >>>>>>>>>>>>>>> clarity when it comes to handling localized date
> > > >>> and
> > > >>>>> time
> > > >>>>>>>>>>>>>>> representations,
> > > >>>>>>>>>>>>>>> and also avoid the inherent confusion of
> > > >>>> java.sql.Date
> > > >>>>>>>>> extending
> > > >>>>>>>>>>>>>>> java.util.Date. Many internal components,
> > > >>>> specifically
> > > >>>>>>>>>>>>>>> Record-oriented
> > > >>>>>>>>>>>>>>> processors and services, rely on date parsing,
> > > >>>> leading
> > > >>>>> to
> > > >>>>>>>>>>>>>>> confusion and
> > > >>>>>>>>>>>>>>> various workarounds. The pattern formats of
> > > >>>>>>> SimpleDateFormat
> > > >>>>>>>>> and
> > > >>>>>>>>>>>>>>> DateTimeFormatter are very similar, but there
> > > >> are a
> > > >>>> few
> > > >>>>>>>> subtle
> > > >>>>>>>>>>>>>>> differences.
> > > >>>>>>>>>>>>>>> Making this transition would provide a much
> > > >> better
> > > >>>>>>> foundation
> > > >>>>>>>>>>>>>>> going forward.
> > > >>>>>>>>>>>>>>> *Conclusion*
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks for giving this proposal some
> > > >> consideration.
> > > >>>>> Many of
> > > >>>>>>>> you
> > > >>>>>>>>>>>>>>> have been
> > > >>>>>>>>>>>>>>> developing NiFi for years and I look forward to
> > > >>> your
> > > >>>>>>>> feedback.
> > > >>>>>>>>> I
> > > >>>>>>>>>>>>>>> would be
> > > >>>>>>>>>>>>>>> glad to put together a more formalized
> > > >>> recommendation
> > > >>>>> on
> > > >>>>>>>>>>>>>>> Confluence and
> > > >>>>>>>>>>>>>>> write up Jira epics if this general approach
> > > >> sounds
> > > >>>>>>> agreeable
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> community.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>>>>> David Handermann
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Ryan Hendrickson <ry...@gmail.com>.
If I put Jira's in, then I'd have to admit I'm a developer vs a user....
:-)
I'm just a power user at heart :-) ...
Ryan
On Thu, Aug 5, 2021 at 2:01 PM Otto Fowler <ot...@gmail.com> wrote:
> Ryan, these are awesome, are there jiras for them? It would be best to
> capture the requirements / use cases
>
> From: Ryan Hendrickson <ry...@gmail.com>
> <ry...@gmail.com>
> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> Date: August 4, 2021 at 01:14:23
> To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
>
> I'll preface with what I'm about to say is not rooted in any real
> understanding of the lift required or other customer experiences... We
> have used NiFi for a long time and are excited for what new features may
> lay ahead in the 2.x line.
>
> ---After having written this, I think our succinct pain points focus
> around large data flow management, auto-scaling container environments, and
> integration of external analytic/enrichment creation.---
>
> Thanks,
> Ryan
>
>
> 1 - Integration with auto-scaling container environments
> We often struggle with scaling custom processors in large flows. We've
> implemented a recent design pattern where we write custom python code
> packaged into a docker image, fronted with a web-service, and deployed into
> a container environment. In NiFi we use InvokeHTTP to call out to the
> web-service. If the load becomes too great, we can easily spin-up more
> than one external web-service without sacrificing the NiFi's overall
> thruput and thread contention. In this scenario, we're using NiFi to
> manage the overall dataflow, routes, ETL services, etc, but not so much
> smarts around auto-scaling processing, etc.
>
> This makes it really easy to receive a dataflow analytic/enrichment from an
> external team, wrap it in a docker image, and deploy it into an
> auto-scaling container environment. This also lets us avoid conversations
> about how performant the code is and if it's meeting a minimum throughput
> requirement - typically XXGB/5minutes.
>
> The request for 2.x would be to make interacting with container deployed
> services a more seamless, streamlined process, possibly through a
> normalized API or as a "Remote Container Environment" that can bind to a
> service with a health endpoint and verify it's alive and well, etc.
>
> 2 - Integration of S3 buckets with Remote Process Groups for outage events
> and data overruns
> Another frequent struggle is the occasional death of a server in a NiFi
> flow. Example -- Our overall NiFi flow consists of ~15 stand-alone NiFis
> and 2 NiFi clusters. The stand-alone NiFi's are linearly chained together
> with Remote Process Groups to perform the overall dataflow goals. If any
> one of the 15 experience an outage, data begins to backup on the previous
> NiFi, hitting backpressure limits, etc. When the down server comes back
> online, it receives a flood of data, often overwhelming it, necessitating
> careful flowfile limits and backpressure points.
>
> The design pattern we're exploring, but would like to see baked into NiFi,
> is that when a Remote Process Group becomes unavailable due to
> server-to-server comms failures AND the backpressure limit is reached on a
> relationship, the Remote Process Group would begin sending data to an S3
> bucket as an automatic failover procedure. When the downed server comes
> back-online, the Remote Process Group port would know to pull data from S3
> AND receive data via the port from the chained server.
>
> 3 - Large Canvas Management
> Beginning with a NiFi flow that consists of ~15 stand-alone NiFis and 2
> NiFi clusters, each canvas is managed individually. To see the entire
> canvas, systems engineering diagrams need to be kept up-to-date identifying
> all the input/output ports. To move processors from server to server often
> requires the verification of the existence of scripts, configuration files,
> NiFi custom nars, and using the templating or Registry to migrate portions
> of the dataflow.
>
> Our current design pattern to accomplish some of these tasks include using
> Ansible, Jenkins, and Nexus to verify script & configuration files are up
> to date. We use Confluence's Gliffy to document the NiFi communication
> paths between the servers.
>
> Ideally, we'd be able to see an overall management view of the Canvases
> together in a single place - it would indicate which server the portions of
> the canvas are deployed on and tie together where the Remote Process Group
> ports are connected. Additionally, NiFi would allow the user to move
> Processors and Process Groups seamlessly from one server to another server
> to help manually distribute and balance data flow processing cycles.
>
>
>
> On Tue, Aug 3, 2021 at 12:45 PM Joe Witt <jo...@gmail.com> wrote:
>
> > Kevin,
> >
> > This may well be a very interesting approach which opens up a lot of
> > options for us in how we tackle a 2.x...
> >
> > Thanks
> >
> > On Tue, Aug 3, 2021 at 8:45 AM Kevin Doran <kd...@gmail.com>
> > wrote:
> > >
> > > Outside of core NiFi, for MiNiFi, I think it would be beneficial to
> > align MiNiFi Java and MiNiFI C++ to use the same method(s) for command
> and
> > control / flow deployment. This was discussed when MiNiFi Java was merged
> > into the NiFi codebase [1].
> > >
> > > My proposal would be to create a Java implementation of the C2 Protocol
> > used by MiNiFi C++, both server-side and client-side, to allow deploying
> to
> > MiNiFi Java instances from a centralize flow definition. Thanks to the
> > unification of MiNiFi Java and NiFi, this could likely also be added as a
> > capability of NiFi at that same time.
> > >
> > > If we are able to do this on the 1.x line, then I would also support
> > deprecating other methods of flow deployment to MiNiFi outlined in [1]
> and
> > removing them in a 2.x release.
> > >
> > > [1] https://github.com/apache/nifi/pull/4933#issuecomment-808411977 <
> > https://github.com/apache/nifi/pull/4933#issuecomment-808411977>
> > > [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design <
> > https://cwiki.apache.org/confluence/display/MINIFI/C2+Design>
> > >
> > > Thanks,
> > > Kevin
> > >
> > > > On Aug 3, 2021, at 10:07 AM, David Handermann <
> > exceptionfactory@gmail.com> wrote:
> > > >
> > > > Thanks for the continued discussion around what can or should be
> > removed.
> > > > Having a clear migration path from version 1 to version 2 will be
> very
> > > > important for supporting current deployments.
> > > >
> > > > Following the example of some other projects, one way to consider the
> > > > upgrade is from the point of view of deprecation warnings. If
> > implemented
> > > > correctly, an installation of NiFi running the latest minor release
> of
> > > > version 1, and showing no deprecation warnings, should be upgradeable
> > to
> > > > version 2 without any changes. Making this work involves accurately
> > tagging
> > > > components and methods with deprecation indicators, and providing a
> > clear
> > > > way to observe deprecation warnings. With the current version
> > containing
> > > > both deprecated components and deprecated methods in various classes,
> > this
> > > > would involve some work to get right. A simple approach might be a
> > named
> > > > logger for deprecation warnings, which would write a separate log
> > file. A
> > > > more advanced approach might involve a tool to analyze the system and
> > flow
> > > > configurations. Either way, it seems that some additional work in a
> > minor
> > > > release version is necessary before considering a major release
> > version.
> > > >
> > > > One other point on compatibility: as long as the core component API
> > > > definition remains backwards compatible, it should be possible to run
> > older
> > > > components. This provides a transition option for customers using
> > > > components such as PostHTTP, or any others that are not actively
> > > > maintained. At some point it may become necessary to break
> > compatibility at
> > > > that level, but at least for version 2, maintaining component API
> > > > compatibility is key.
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Sun, Aug 1, 2021 at 10:23 AM Mark Bean <ma...@gmail.com>
> > wrote:
> > > >
> > > >> I created a JIRA ticket to investigate and improve the performance
> of
> > > >> InvokeHTTP. It includes a flow definition for benchmarking the
> > performance
> > > >> of both PostHTTP and InvokeHTTP.
> > > >>
> > > >> https://issues.apache.org/jira/browse/NIFI-8968
> > > >>
> > > >> Thanks,
> > > >> Mark
> > > >>
> > > >> On Fri, Jul 30, 2021 at 6:41 PM Adam Taft <ad...@adamtaft.com>
> wrote:
> > > >>
> > > >>> I'm not seeing the side thread that was going to discuss
> deprecation
> > of
> > > >>> PostHTTP. Has that thread started and I just don't see it?
> > > >>>
> > > >>> One (significant?) concern with PostHTTP is the smooth integration
> of
> > > >>> NiFi-to-NiFi communication that is very transparently enabled with
> > the
> > > >>> ListenHTTP and PostHTTP processors. There's some special logic in
> > there
> > > >> for
> > > >>> handling flowfiles that InvokeHTTP doesn't really (nor should
> really)
> > > >> have.
> > > >>>
> > > >>> I know of several (large) NiFi installations that rely on the
> > PostHTTP /
> > > >>> ListenHTTP combination. It has enabled NiFi to NiFi communication
> for
> > > >> folks
> > > >>> reluctant or unable to enable site-to-site type configuration.
> > > >>>
> > > >>> Honestly, I don't know why we'd want to "deprecate" any processor,
> as
> > > >>> opposed to just moving it to a new location. If these processors
> can
> > be
> > > >>> ported and maintained to whatever the 2.0 API looks like, there's
> > > >> possibly
> > > >>> little harm keeping them around.
> > > >>>
> > > >>> And by the way, what happened to the "marketplace" concept? Is this
> > being
> > > >>> considered for 2.0 as well? Because relocating the deprecated
> > processors
> > > >>> to an external nar might be the best solution. Losing PostHTTP
> > entirely I
> > > >>> think would be a mistake, but I'd conceptually support relocating
> it.
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> /Adam
> > > >>>
> > > >>> On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <jo...@gmail.com>
> wrote:
> > > >>>
> > > >>>> Looks like we just need to knock out 5 JIRAs :) [1]
> > > >>>>
> > > >>>> I felt like we had a label folks were using at one point but
> quickly
> > > >>>> looking revealed nothing exciting. After this confluence page
> > > >>>> stabilizes a bit we can probably knock out some JIRAs and such.
> > > >>>>
> > > >>>> [1]
> https://issues.apache.org/jira/projects/NIFI/versions/12339599
> > > >>>>
> > > >>>> On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <
> > ottobackwards@gmail.com>
> > > >>>> wrote:
> > > >>>>>
> > > >>>>> I find myself wishing I had a list of all the jiras / issues that
> > > >> have
> > > >>>>> been put off for a 2.0 release because they required some change
> or
> > > >>>> another
> > > >>>>> :(
> > > >>>>>
> > > >>>>> From: Joe Witt <jo...@gmail.com> <jo...@gmail.com>
> > > >>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > >> dev@nifi.apache.org>
> > > >>>>> Date: July 27, 2021 at 12:30:35
> > > >>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
> dev@nifi.apache.org
> > >
> > > >>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > >>>>>
> > > >>>>> A few thoughts:
> > > >>>>>
> > > >>>>> 1. I would love to see deprecation notices show up in the UI in
> > > >>>>> various ways to help motivate users to move off things to more
> > > >>>>> supportable things. That is not a prerequisite for anything
> > happening
> > > >>>>> however. Just a good feature/nice thing to do for users when
> > someone
> > > >>>>> is able to tackle it.
> > > >>>>>
> > > >>>>> 2. The decision to deprecate something and to further remove it
> > need
> > > >>>>> not mean there is a superior solution available. If that thing
> > itself
> > > >>>>> isn't getting the love/attention it needs to be
> > > >>>>> maintained/supported/healthy going forward that alone is enough
> to
> > > >>>>> remove it. That might well be the case with PostHTTP [1] and for
> > > >>>>> comparison you can see how much effort has gone into InvokeHTTP
> > [2].
> > > >>>>>
> > > >>>>> 3. When discussing a 2.0 release each thing we add as a 'must do'
> > the
> > > >>>>> further away from reality such a release will become. We'll have
> to
> > > >>>>> get very specific about 'musts' vs 'wants'.
> > > >>>>>
> > > >>>>> [1]
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> >
>
> https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
> > > >>>>> [2]
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> >
>
> https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
> > > >>>>>
> > > >>>>> On Tue, Jul 27, 2021 at 8:47 AM David Handermann
> > > >>>>> <ex...@apache.org> wrote:
> > > >>>>>>
> > > >>>>>> Thanks Mark, providing a template or comparison statistics with
> > > >> Java
> > > >>>>>> versions and component configuration details would be very
> > helpful.
> > > >>> If
> > > >>>> it
> > > >>>>>> is possible to run tests using a public API or deployable
> service,
> > > >>> that
> > > >>>>>> would also help confirm potential differences.
> > > >>>>>>
> > > >>>>>> Showing a deprecation notice in the UI could be helpful, perhaps
> > > >> as a
> > > >>>>>> configurable option. NIFI-8650 describes a general Flow Analysis
> > > >>>>>> capability, and it seems like that might be one possible way to
> > > >>> surface
> > > >>>>>> deprecation warnings. For something more specific to component
> > > >>>>> deprecation,
> > > >>>>>> it seems important to find a balance between making it obvious
> and
> > > >>>> making
> > > >>>>>> it something that ends up getting ignored.
> > > >>>>>>
> > > >>>>>> Regards,
> > > >>>>>> David Handermann
> > > >>>>>>
> > > >>>>>> On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <
> mark.o.bean@gmail.com
> > >
> > > >>>> wrote:
> > > >>>>>>
> > > >>>>>>> I'll start a new thread for PostHTTP when I get a template
> and/or
> > > >>>>> detailed
> > > >>>>>>> stats.
> > > >>>>>>>
> > > >>>>>>> I know the deprecation is noted in the documentation. That's a
> > > >>>>> necessary
> > > >>>>>>> and minimum level of notification. I was suggesting it be more
> > > >>>> obvious
> > > >>>>> in
> > > >>>>>>> the UI. I think it would be beneficial to somehow be aware of
> the
> > > >>>>>>> deprecation status simply by looking at the flow (perhaps even
> on
> > > >>> the
> > > >>>>>>> summary pages too), and without having to open the
> documentation
> > > >>> for
> > > >>>>> every
> > > >>>>>>> processor to confirm whether or not a component is marked as
> > > >>>>> deprecated.
> > > >>>>>>>
> > > >>>>>>> Thanks,
> > > >>>>>>> Mark
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
> > > >>>>>>> exceptionfactory@apache.org> wrote:
> > > >>>>>>>
> > > >>>>>>>> Mark,
> > > >>>>>>>>
> > > >>>>>>>> Thanks for the feedback. It may be better to start a separate
> > > >>>> thread
> > > >>>>> on
> > > >>>>>>>> PostHTTP, but can you provide an example flow demonstrating
> the
> > > >>>>>>> performance
> > > >>>>>>>> differences between PostHTTP and InvokeHTTP?
> > > >>>>>>>>
> > > >>>>>>>> PostHTTP uses the Apache HttpComponents library, whereas
> > > >>> InvokeHTTP
> > > >>>>> uses
> > > >>>>>>>> OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates
> > > >> for
> > > >>>>> OkHttp,
> > > >>>>>>>> so it would be important to test with the most recent version.
> > > >> It
> > > >>>> is
> > > >>>>> also
> > > >>>>>>>> worth noting that test classes for PostHTTP are now
> integration
> > > >>>> tests
> > > >>>>> as
> > > >>>>>>>> opposed to unit tests, which means they are not executed as
> > > >> part
> > > >>> of
> > > >>>>> the
> > > >>>>>>>> automated builds.
> > > >>>>>>>>
> > > >>>>>>>> The NiFi documentation includes the contents of the
> > > >>>> DeprecationNotice
> > > >>>>> for
> > > >>>>>>>> PostHTTP:
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> >
>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
> > > >>>>>>>>
> > > >>>>>>>> Regards,
> > > >>>>>>>> David Handermann
> > > >>>>>>>>
> > > >>>>>>>> On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <
> > > >> mark.o.bean@gmail.com
> > > >>>>
> > > >>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> I'm strongly in favor of reducing tech debt, and moving
> > > >>>>> deliberately
> > > >>>>>>> to a
> > > >>>>>>>>> 2.0 release. I have a concern with just one processor that is
> > > >>>>> currently
> > > >>>>>>>>> marked as deprecated: PostHTTP. (I have not evaluated
> > > >>>> specifically
> > > >>>>> any
> > > >>>>>>>>> other deprecated components; I cannot say if there are or are
> > > >>> not
> > > >>>>>>> similar
> > > >>>>>>>>> issues elsewhere.) I understand the rationale for deprecating
> > > >>>> this
> > > >>>>>>>>> processor in that it eliminates a processor whose
> > > >> functionality
> > > >>>> is
> > > >>>>>>>>> available elsewhere, specifically in the more flexible
> > > >>>> InvokeHTTP.
> > > >>>>>>>> However,
> > > >>>>>>>>> in my experience and testing, PostHTTP performs transfers
> > > >> with
> > > >>>> far
> > > >>>>>>>> greater
> > > >>>>>>>>> throughput than InvokeHTTP. I would not be in favor of
> > > >> removing
> > > >>>>>>> PostHTTP
> > > >>>>>>>>> unless/until InvokeHTTP is refactored to increase its
> > > >>> throughput
> > > >>>>>>>>> capability.
> > > >>>>>>>>>
> > > >>>>>>>>> Has anyone else continued to use PostHTTP over InvokeHTTP for
> > > >>>>> similar
> > > >>>>>>>>> reasons? Or, is there a performance-related configuration for
> > > >>>>>>> InvokeHTTP
> > > >>>>>>>> I
> > > >>>>>>>>> may have missed?
> > > >>>>>>>>>
> > > >>>>>>>>> Also, in order to help facilitate a smooth transition to 2.0
> > > >>>> from a
> > > >>>>>>> user
> > > >>>>>>>>> perspective, would it be advisable to add some sort of visual
> > > >>>>> indicator
> > > >>>>>>>> in
> > > >>>>>>>>> the UI for components that are currently annotated as
> > > >>>> @Deprecated?
> > > >>>>> This
> > > >>>>>>>>> might help users proactively modify their flow prior to a
> > > >>> release
> > > >>>>> that
> > > >>>>>>>>> would otherwise break it.
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <
> > > >> bbende@gmail.com>
> > > >>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> With the merging of MiNiFi and registry into the NiFi repo,
> > > >>>> we've
> > > >>>>>>>>>> actually gone more towards a mono-repo where everything is
> > > >>>>> released
> > > >>>>>>>>>> together. Eventually it would still be nice to have a
> > > >> smaller
> > > >>>>> base
> > > >>>>>>>>>> distribution containing just the framework and standard
> > > >> NARs,
> > > >>>> but
> > > >>>>> it
> > > >>>>>>>>>> is hard to tackle that until we provide a way for users to
> > > >>>> easily
> > > >>>>>>>>>> obtain the NARs in some other way.
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <
> > > >>>>> edward.armes@gmail.com
> > > >>>>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Given the major version number shift and the spliting up
> > > >> of
> > > >>>>>>>> processors
> > > >>>>>>>>>> into
> > > >>>>>>>>>>> multiple NAR's. I'd like to suggest that we start
> > > >>>> individually
> > > >>>>>>>>> versioning
> > > >>>>>>>>>>> individual NARs/ bundles.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I can see this bringing a large number of benifits
> > > >>> including
> > > >>>>> making
> > > >>>>>>>>> Nifi
> > > >>>>>>>>>>> more deployable with things RPM, but also potentially
> > > >>>> allowing
> > > >>>>>>> those
> > > >>>>>>>>> that
> > > >>>>>>>>>>> have to deploy managed Nifi instances easier to upgrade.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Edward
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Mon, 26 Jul 2021, 20:42 Otto Fowler, <
> > > >>>> ottobackwards@gmail.com>
> > > >>>>>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> The issue with updating the aws sdk is if it breaks any
> > > >>> one
> > > >>>>> of
> > > >>>>>>> the
> > > >>>>>>>>>>>> processors.
> > > >>>>>>>>>>>> the Web Gateway API invoke processor for example is not
> > > >>>> using
> > > >>>>> a
> > > >>>>>>>> high
> > > >>>>>>>>>> level
> > > >>>>>>>>>>>> purpose build client and may break.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> If we change the aws version, we need to coordinate in
> > > >>>> such a
> > > >>>>> way
> > > >>>>>>>>> that
> > > >>>>>>>>>> they
> > > >>>>>>>>>>>> all
> > > >>>>>>>>>>>> can come along reasonably.
> > > >>>>>>>>>>>> IE: what happens if 1 or 2 break but the rest or OK?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> From: David Handermann <ex...@apache.org>
> > > >>>>>>>>>>>> <ex...@apache.org>
> > > >>>>>>>>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > >>>>>>>>> dev@nifi.apache.org>
> > > >>>>>>>>>>>> Date: July 26, 2021 at 09:33:42
> > > >>>>>>>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > >>>>>>> dev@nifi.apache.org
> > > >>>>>>>>>
> > > >>>>>>>>>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Chris,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks for the reply and recommendations. It seems like
> > > >>>> some
> > > >>>>> of
> > > >>>>>>> the
> > > >>>>>>>>>> work to
> > > >>>>>>>>>>>> reorganize the module structure could be done outside
> > > >> of
> > > >>> a
> > > >>>>> major
> > > >>>>>>>>>> release,
> > > >>>>>>>>>>>> but it would be great to target any breaking changes
> > > >> for
> > > >>>> 2.0.
> > > >>>>>>>>> Perhaps a
> > > >>>>>>>>>>>> separate feature proposal on module restructuring, with
> > > >>> the
> > > >>>>> goal
> > > >>>>>>> of
> > > >>>>>>>>>>>> supporting optimized builds, would be a helpful way to
> > > >>> move
> > > >>>>> that
> > > >>>>>>>> part
> > > >>>>>>>>>> of
> > > >>>>>>>>>>>> the discussion forward.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Regarding updating AWS SDK to version 2, it seems like
> > > >>> that
> > > >>>>> might
> > > >>>>>>>> be
> > > >>>>>>>>>>>> possible now. I haven't taken a close look at the
> > > >>>> referencing
> > > >>>>>>>>>> components,
> > > >>>>>>>>>>>> so I'm not sure about the level of effort involved.
> > > >> Minor
> > > >>>>> NiFi
> > > >>>>>>>>> version
> > > >>>>>>>>>>>> updates have incorporated major new versions of
> > > >>>> dependencies.
> > > >>>>> For
> > > >>>>>>>>>> example,
> > > >>>>>>>>>>>> NiFi 1.14 included an upgrade from Spring Framework 4
> > > >> to
> > > >>> 5.
> > > >>>>> On
> > > >>>>>>> the
> > > >>>>>>>>> one
> > > >>>>>>>>>>>> hand, including the AWS SDK update as part of a major
> > > >>>> release
> > > >>>>>>> seems
> > > >>>>>>>>>>>> helpful, but unless there are changes that break
> > > >> existing
> > > >>>>>>> component
> > > >>>>>>>>>>>> properties, upgrading the AWS SDK could be worked
> > > >>>>> independently.
> > > >>>>>>>>>> Others may
> > > >>>>>>>>>>>> have more insight into particular usage of that
> > > >> library.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>> David Handermann
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > > >>>>>>>>>>>> <ch...@naimuri.com.invalid> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Might be worth considering refactoring the build as
> > > >>> part
> > > >>>> of
> > > >>>>>>> this
> > > >>>>>>>>> work
> > > >>>>>>>>>>>> too,
> > > >>>>>>>>>>>>> e.g. only building the bits of the repo affected by a
> > > >>>>> commit,
> > > >>>>>>>> etc.
> > > >>>>>>>>> -
> > > >>>>>>>>>>>>> discussed briefly in previous threads but don't think
> > > >>> any
> > > >>>>>>> changes
> > > >>>>>>>>>> made
> > > >>>>>>>>>>>> yet.
> > > >>>>>>>>>>>>> If NARs/components are likely to be split up and
> > > >>>> refactored
> > > >>>>>>> then
> > > >>>>>>>>> such
> > > >>>>>>>>>>>> work
> > > >>>>>>>>>>>>> around the build probably makes sense to consider.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I've a couple of PRs open that include updates to
> > > >>>>> Elasticsearch
> > > >>>>>>>>>> versions
> > > >>>>>>>>>>>>> already, although I stopped at 7.10.2 (after which
> > > >>>> Elastic
> > > >>>>>>>> changed
> > > >>>>>>>>>>>> licence)
> > > >>>>>>>>>>>>> in case there were licence concerns. But more work
> > > >> can
> > > >>> be
> > > >>>>> done
> > > >>>>>>> to
> > > >>>>>>>>>> tidy up
> > > >>>>>>>>>>>>> the processors, absolutely.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> AWS libraries to v2 would seem a sensible move and a
> > > >>>>> refactor
> > > >>>>>>> of
> > > >>>>>>>>>> those
> > > >>>>>>>>>>>>> processors as well.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Chris Sampson
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > > >>>>>>>>>>>> exceptionfactory@apache.org>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks for pointing out the standard NAR bundles,
> > > >>> Mark.
> > > >>>>> There
> > > >>>>>>>>> are a
> > > >>>>>>>>>>>>> number
> > > >>>>>>>>>>>>>> of components in the standard NAR bundles with
> > > >>>> particular
> > > >>>>>>>>>> dependencies
> > > >>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>> would make more sense in separate NARs.
> > > >> Reorganizing
> > > >>>> the
> > > >>>>>>>> standard
> > > >>>>>>>>>> NAR
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>> components with limited dependencies and wide
> > > >>>>> applicability
> > > >>>>>>>> would
> > > >>>>>>>>>>>>>> definitely help with future maintenance.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>>>> David Handermann
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> > > >>>>>>>>> markap14@hotmail.com>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> There’s also some code that exists in order to
> > > >>>> maintain
> > > >>>>>>>>> backward
> > > >>>>>>>>>>>>>>> compatibility in the repositories. I would very
> > > >>> much
> > > >>>>> like
> > > >>>>>>> the
> > > >>>>>>>>>>>>>> repositories
> > > >>>>>>>>>>>>>>> to contain no unnecessary code. And swap file
> > > >>> format
> > > >>>>>>> supports
> > > >>>>>>>>>> really
> > > >>>>>>>>>>>>> old
> > > >>>>>>>>>>>>>>> formats. And the old impls of the repositories
> > > >>>>> themselves,
> > > >>>>>>>> like
> > > >>>>>>>>>>>>>>> PersistentProvRepo instead of WriteAheadProv
> > > >> Repo,
> > > >>>> etc.
> > > >>>>>>> Lots
> > > >>>>>>>> of
> > > >>>>>>>>>> stuff
> > > >>>>>>>>>>>>>> there
> > > >>>>>>>>>>>>>>> that can be removed. And some methods in
> > > >>>> ProcessSession
> > > >>>>>>> that
> > > >>>>>>>>> are
> > > >>>>>>>>>>>> never
> > > >>>>>>>>>>>>>> used
> > > >>>>>>>>>>>>>>> by any processor in the codebase but exists in
> > > >> the
> > > >>>>> public
> > > >>>>>>> API
> > > >>>>>>>>> so
> > > >>>>>>>>>>>> can’t
> > > >>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>> removed till 2.0.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I think his is also a great time to clean up the
> > > >>>>> “standard
> > > >>>>>>>>> nar.”
> > > >>>>>>>>>> At
> > > >>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>> point, it’s something like 70 MB. And many of the
> > > >>>>>>> components
> > > >>>>>>>>>> there
> > > >>>>>>>>>>>> are
> > > >>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>> really “standard” - things like connecting to
> > > >> FTP &
> > > >>>>> SFTP
> > > >>>>>>>>>> servers, XML
> > > >>>>>>>>>>>>>>> processing, Jolt transform, etc. could
> > > >> potentially
> > > >>> be
> > > >>>>> moved
> > > >>>>>>>>> into
> > > >>>>>>>>>>>> other
> > > >>>>>>>>>>>>>>> nars. The
> > > >>>>> nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
> > > >>>>>>> is
> > > >>>>>>>>>> 6.9 MB
> > > >>>>>>>>>>>> is
> > > >>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>> necessary for stateless or minifi java. Lots of
> > > >>>> things
> > > >>>>>>>> probably
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>>>>> reconsider within the standard nar.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I definitely think this is a reasonable approach,
> > > >>> to
> > > >>>>> allow
> > > >>>>>>>> for
> > > >>>>>>>>> a
> > > >>>>>>>>>> 2.0
> > > >>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>> is not a huge feature release but allows the
> > > >>> project
> > > >>>> to
> > > >>>>> be
> > > >>>>>>>>>> simpler
> > > >>>>>>>>>>>> and
> > > >>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>> nimble.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks
> > > >>>>>>>>>>>>>>> -Mark
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > > >>>>>>>>>> mikerthomsen@gmail.com
> > > >>>>>>>>>>>>>> <mailto:
> > > >>>>>>>>>>>>>>> mikerthomsen@gmail.com>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Russell,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> AFAICT from looking at Elastic's repos, the low
> > > >>> level
> > > >>>>> REST
> > > >>>>>>>>>> client is
> > > >>>>>>>>>>>>>>> still fine.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> >
>
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Our Elasticsearch support is spread over two NARs
> > > >>> at
> > > >>>>>>> present.
> > > >>>>>>>>> One
> > > >>>>>>>>>>>> uses
> > > >>>>>>>>>>>>>>> OkHttp and the other uses that low level Elastic
> > > >>> REST
> > > >>>>>>> client.
> > > >>>>>>>>>>>>>>> Therefore, I think we're fine on licensing for
> > > >> the
> > > >>>>> moment.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Mike
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > > >>>>>>>>>>>> russ@windofkeltia.com
> > > >>>>>>>>>>>>>>> <ma...@windofkeltia.com>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Bringing up Elastic also reminds me that the
> > > >>> Elastic
> > > >>>>>>>> framework
> > > >>>>>>>>>> has
> > > >>>>>>>>>>>> just
> > > >>>>>>>>>>>>>>> recently transitioned out of Open Source, so to
> > > >>>>> acknowledge
> > > >>>>>>>>> that,
> > > >>>>>>>>>>>> maybe
> > > >>>>>>>>>>>>>>> some effort toward OpenSearch--I say this not
> > > >>>>> understanding
> > > >>>>>>>>>> exactly
> > > >>>>>>>>>>>> how
> > > >>>>>>>>>>>>>>> this sort of thing is considered in a
> > > >> large-scale,
> > > >>>>>>>> world-class
> > > >>>>>>>>>>>> software
> > > >>>>>>>>>>>>>>> project like Apache NiFi. (I'm not a contributor,
> > > >>>> just
> > > >>>>> a
> > > >>>>>>>>> grateful
> > > >>>>>>>>>>>>>>> consumer.)
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Russ
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > >>>>>>>>>>>>>>> Along with the itemized list for ancient
> > > >> components
> > > >>>> we
> > > >>>>>>> should
> > > >>>>>>>>>> look at
> > > >>>>>>>>>>>>>>> updating versions of drivers, SDKs, etc. for
> > > >>> external
> > > >>>>>>> systems
> > > >>>>>>>>>> such as
> > > >>>>>>>>>>>>>>> Elasticsearch, Cassandra, etc. There may be
> > > >>> breaking
> > > >>>>>>> changes
> > > >>>>>>>>> but
> > > >>>>>>>>>> 2.0
> > > >>>>>>>>>>>>>>> is probably the right time to get things up to
> > > >> date
> > > >>>> to
> > > >>>>> make
> > > >>>>>>>>> them
> > > >>>>>>>>>> more
> > > >>>>>>>>>>>>>>> useful to more people.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > > >>>>>>>>>> thenatog@gmail.com
> > > >>>>>>>>>>>>>> <mailto:
> > > >>>>>>>>>>>>>>> thenatog@gmail.com>> wrote:
> > > >>>>>>>>>>>>>>> I'm a +1 for removing pretty much all of this
> > > >>> stuff.
> > > >>>>> There
> > > >>>>>>>> are
> > > >>>>>>>>>>>> security
> > > >>>>>>>>>>>>>>> implications to keeping old dependencies around,
> > > >> so
> > > >>>> the
> > > >>>>>>> more
> > > >>>>>>>>> old
> > > >>>>>>>>>> code
> > > >>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>> can remove the better. I agree that eventually we
> > > >>>> need
> > > >>>>> to
> > > >>>>>>>> move
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>>>> supporting only Java 11+, and as our next release
> > > >>>> will
> > > >>>>>>>> probably
> > > >>>>>>>>>> be
> > > >>>>>>>>>>>>> about
> > > >>>>>>>>>>>>>> 4
> > > >>>>>>>>>>>>>>> - 6 months from now that doesn't seem too soon.
> > > >> We
> > > >>>>> could
> > > >>>>>>>>>> potentially
> > > >>>>>>>>>>>>>> break
> > > >>>>>>>>>>>>>>> this in two and remove the deprecated processors
> > > >>> and
> > > >>>>> leave
> > > >>>>>>>> 1.x
> > > >>>>>>>>> on
> > > >>>>>>>>>>>> Java
> > > >>>>>>>>>>>>> 8,
> > > >>>>>>>>>>>>>>> and finally start on 2.x which would support only
> > > >>>> Java
> > > >>>>> 11.
> > > >>>>>>>> I'm
> > > >>>>>>>>>> unsure
> > > >>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>> what implications changing the date and time
> > > >>> handling
> > > >>>>> would
> > > >>>>>>>>> have
> > > >>>>>>>>>> -
> > > >>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>> running systems that use long term historical
> > > >> logs,
> > > >>>>>>>> unexpected
> > > >>>>>>>>>>>> impacts
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> time logging could be a problem.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> As Joe says I think feature work will have to be
> > > >>>>> dedicated
> > > >>>>>>> to
> > > >>>>>>>>>> 2.x and
> > > >>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>> could support 1.x for security fixes for some
> > > >>> period
> > > >>>> of
> > > >>>>>>> time.
> > > >>>>>>>>> 2.x
> > > >>>>>>>>>>>> seems
> > > >>>>>>>>>>>>>>> like a gargantuan task but it's probably time to
> > > >>> get
> > > >>>>>>> started.
> > > >>>>>>>>> Not
> > > >>>>>>>>>>>> sure
> > > >>>>>>>>>>>>>> how
> > > >>>>>>>>>>>>>>> we handle all open PRs and the transition between
> > > >>> 1.x
> > > >>>>> and
> > > >>>>>>>> 2.x.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> > > >>>>>>>> joe.witt@gmail.com
> > > >>>>>>>>>>>> <mailto:
> > > >>>>>>>>>>>>>>> joe.witt@gmail.com>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Jon
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> You're right we have to be careful and you're
> > > >> right
> > > >>>>> there
> > > >>>>>>> are
> > > >>>>>>>>>> still
> > > >>>>>>>>>>>>>>> significant Java 8 users out there. But we also
> > > >>> have
> > > >>>> to
> > > >>>>> be
> > > >>>>>>>>>> careful
> > > >>>>>>>>>>>>>>> about security and sustainability of the
> > > >> codebase.
> > > >>> If
> > > >>>>> we
> > > >>>>>>> had
> > > >>>>>>>>>> talked
> > > >>>>>>>>>>>>>>> about this last year when that article came out
> > > >> I'd
> > > >>>>> have
> > > >>>>>>>> agreed
> > > >>>>>>>>>> it is
> > > >>>>>>>>>>>>>>> too early. Interestingly that link seems to get
> > > >>>> updated
> > > >>>>>>> and I
> > > >>>>>>>>>> tried
> > > >>>>>>>>>>>>>>> [1] and found more recent data (not sure how
> > > >>> recent).
> > > >>>>>>> Anyway
> > > >>>>>>>> it
> > > >>>>>>>>>>>>>>> suggests Java 8 is still the top dog but we see
> > > >>> good
> > > >>>>> growth
> > > >>>>>>>> on
> > > >>>>>>>>>> 11. In
> > > >>>>>>>>>>>>>>> my $dayjob this aligns to what I'm seeing too.
> > > >>>>> Customers
> > > >>>>>>>> didn't
> > > >>>>>>>>>> seem
> > > >>>>>>>>>>>>>>> to care about Java 11 until later half last year
> > > >>> and
> > > >>>>> now
> > > >>>>>>>>>> suddenly it
> > > >>>>>>>>>>>>>>> is all over the place.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I think once we put out a NiFi 2.0 release we'd
> > > >> see
> > > >>>>> rapid
> > > >>>>>>>>>> decrease in
> > > >>>>>>>>>>>>>>> work on the 1.x line just being blunt. We did
> > > >> this
> > > >>>> many
> > > >>>>>>> years
> > > >>>>>>>>> ago
> > > >>>>>>>>>>>>>>> with 0.x to 1.x and we stood behind 0.x for a
> > > >> while
> > > >>>>> (maybe
> > > >>>>>>> a
> > > >>>>>>>>>> year or
> > > >>>>>>>>>>>>>>> so) but it was purely bug fix/security related
> > > >>> bits.
> > > >>>> We
> > > >>>>>>> would
> > > >>>>>>>>>> need to
> > > >>>>>>>>>>>>>>> do something similar. But feature work would
> > > >> almost
> > > >>>>>>> certainly
> > > >>>>>>>>> go
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>>>>> the 2.x line. Maybe there are other workable
> > > >> models
> > > >>>> but
> > > >>>>> my
> > > >>>>>>>>>> instinct
> > > >>>>>>>>>>>>>>> suggests this is likely to follow a similar path.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> ...anyway I agree it isn't that easy of a call to
> > > >>>> dump
> > > >>>>> Java
> > > >>>>>>>> 8.
> > > >>>>>>>>> We
> > > >>>>>>>>>>>>>>> need to make the call in both the interests of
> > > >> the
> > > >>>> user
> > > >>>>>>> base
> > > >>>>>>>>> and
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>>> contributor base of the community.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> [1]
> > > >>>> https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks
> > > >>>>>>>>>>>>>>> Joe
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
> > > >>>>>>> joe.witt@gmail.com
> > > >>>>>>>>>> <mailto:
> > > >>>>>>>>>>>>>>> joe.witt@gmail.com>> wrote:
> > > >>>>>>>>>>>>>>> Russ
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Yeah the flow registry is a key part of it. But
> > > >>> also
> > > >>>>> now
> > > >>>>>>> you
> > > >>>>>>>>> can
> > > >>>>>>>>>>>>>>> download the flow definition in JSON (upload i
> > > >>> think
> > > >>>> is
> > > >>>>>>> there
> > > >>>>>>>>> now
> > > >>>>>>>>>>>>>>> too). Templates offered a series of challenges
> > > >> such
> > > >>>> as
> > > >>>>> we
> > > >>>>>>>> store
> > > >>>>>>>>>> them
> > > >>>>>>>>>>>>>>> in the flow definition which has made flows
> > > >> massive
> > > >>>> in
> > > >>>>> an
> > > >>>>>>>>>> unintended
> > > >>>>>>>>>>>>>>> way which isn't fun for cluster behavior.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> We have a couple cases where we headed down a
> > > >>>>> particular
> > > >>>>>>>>> concept
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>>>>> came up with better approaches later. We need to
> > > >>>>> reconcile
> > > >>>>>>>>> these
> > > >>>>>>>>>> with
> > > >>>>>>>>>>>>>>> the benefit of hindsight, and while being careful
> > > >>> to
> > > >>>> be
> > > >>>>> not
> > > >>>>>>>>>> overly
> > > >>>>>>>>>>>>>>> disruptive to existing users, to reduce the
> > > >>>>>>>>> codebase/maintenance
> > > >>>>>>>>>>>>>>> burden and allow continued evolution of the
> > > >>> project.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > > >>>>>>>>>>>> russ@windofkeltia.com
> > > >>>>>>>>>>>>>>> <ma...@windofkeltia.com>>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>> Joe,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I apologize for the off-topic intrusion, but what
> > > >>>>> replaces
> > > >>>>>>>>>> templates?
> > > >>>>>>>>>>>>>>> The Registry? Templates rocked and we have used
> > > >>> them
> > > >>>>> since
> > > >>>>>>>>> 0.5.x.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Russ
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On 7/23/21 8:31 AM, Joe Witt wrote:
> > > >>>>>>>>>>>>>>> David,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I think this is a highly reasonable approach and
> > > >>>> such a
> > > >>>>>>> focus
> > > >>>>>>>>>> will
> > > >>>>>>>>>>>>>>> greatly help make a 2.0 release far more
> > > >>> approachable
> > > >>>>> to
> > > >>>>>>>> knock
> > > >>>>>>>>>> out.
> > > >>>>>>>>>>>>>>> Not only that but tech debt reduction would help
> > > >>> make
> > > >>>>> work
> > > >>>>>>>>>> towards
> > > >>>>>>>>>>>>>>> major features we'd think about in a 'major
> > > >>> release'
> > > >>>>> sense
> > > >>>>>>>> more
> > > >>>>>>>>>>>>>>> approachable.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> We should remove all deprecated things (as well
> > > >> as
> > > >>>>> verify
> > > >>>>>>> we
> > > >>>>>>>>>> have the
> > > >>>>>>>>>>>>>>> right list). We should remove/consider removal of
> > > >>>>>>> deprecated
> > > >>>>>>>>>>>>>>> concepts
> > > >>>>>>>>>>>>>>> like templates. We should consider whether we can
> > > >>>>> resolve
> > > >>>>>>> the
> > > >>>>>>>>>>>>>>> various
> > > >>>>>>>>>>>>>>> ways we've handled what are now parameters down
> > > >> to
> > > >>>> one
> > > >>>>>>> clean
> > > >>>>>>>>>>>>>>> approach.
> > > >>>>>>>>>>>>>>> We should remove options in the nifi.properties
> > > >>> which
> > > >>>>> turn
> > > >>>>>>>> out
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>>>> never be used quite right (if there are). There
> > > >> is
> > > >>>>> quite a
> > > >>>>>>>> bit
> > > >>>>>>>>> we
> > > >>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>> do purely in the name of tech debt reduction.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Lots to consider here but I think this is the
> > > >> right
> > > >>>>>>>> discussion.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Than ks
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> > > >>>>>>>> bbende@gmail.com
> > > >>>>>>>>>>>> <mailto:
> > > >>>>>>>>>>>>>>> bbende@gmail.com>>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>> I'm a +1 for this... Not sure if this falls under
> > > >>>>> "Removing
> > > >>>>>>>>>>>>>>> Deprecated
> > > >>>>>>>>>>>>>>> Components", but I think we should also look at
> > > >>>>> anything
> > > >>>>>>> that
> > > >>>>>>>>> has
> > > >>>>>>>>>>>>>>> been
> > > >>>>>>>>>>>>>>> marked as deprecated throughout the code base as
> > > >> a
> > > >>>>>>> candidate
> > > >>>>>>>>> for
> > > >>>>>>>>>>>>>>> removal. There are quite a few classes, methods,
> > > >>>>>>> properties,
> > > >>>>>>>>> etc
> > > >>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>> have been waiting for a chance to be removed.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > >>>>>>>>>>>>>>> <exceptionfactory@apache.org<mailto:
> > > >>>>>>>>> exceptionfactory@apache.org
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>> Team,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> With all of the excellent work that many have
> > > >>>>> contributed
> > > >>>>>>> to
> > > >>>>>>>>> NiFi
> > > >>>>>>>>>>>>>>> over the
> > > >>>>>>>>>>>>>>> years, the code base has also accumulated some
> > > >>> amount
> > > >>>>> of
> > > >>>>>>>>>> technical
> > > >>>>>>>>>>>>>>> debt. A
> > > >>>>>>>>>>>>>>> handful of components have been marked as
> > > >>> deprecated,
> > > >>>>> and
> > > >>>>>>>> some
> > > >>>>>>>>>>>>>>> components
> > > >>>>>>>>>>>>>>> remain in the code base to support integration
> > > >> with
> > > >>>> old
> > > >>>>>>>>> versions
> > > >>>>>>>>>>>>>>> of various
> > > >>>>>>>>>>>>>>> products. Following the principles of semantic
> > > >>>>> versioning,
> > > >>>>>>>>>>>>>>> introducing a
> > > >>>>>>>>>>>>>>> major release would provide the opportunity to
> > > >>> remove
> > > >>>>> these
> > > >>>>>>>>>>>>>>> deprecated and
> > > >>>>>>>>>>>>>>> unsupported components.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Rather than focusing the next major release on
> > > >> new
> > > >>>>>>> features,
> > > >>>>>>>>> what
> > > >>>>>>>>>>>>>>> do you
> > > >>>>>>>>>>>>>>> think about focusing on technical debt removal?
> > > >>> This
> > > >>>>>>> approach
> > > >>>>>>>>>>>>>>> would not
> > > >>>>>>>>>>>>>>> make for the most interesting release, but it
> > > >>>> provides
> > > >>>>> the
> > > >>>>>>>>>>>>>>> opportunity to
> > > >>>>>>>>>>>>>>> clean up elements that involve breaking changes.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Focusing on technical debt, at least three
> > > >> primary
> > > >>>>> goals
> > > >>>>>>> come
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>>>> mind for
> > > >>>>>>>>>>>>>>> the next major release:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> 1. Removal of deprecated and unmaintained
> > > >>> components
> > > >>>>>>>>>>>>>>> 2. Require Java 11 as the minimum supported
> > > >> version
> > > >>>>>>>>>>>>>>> 3. Transition internal date and time handling to
> > > >>> JSR
> > > >>>>> 310
> > > >>>>>>>>>> java.time
> > > >>>>>>>>>>>>>>> components
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> *Removing Deprecated Components*
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Removing support for older and deprecated
> > > >>> components
> > > >>>>>>>> provides a
> > > >>>>>>>>>>>>>>> great
> > > >>>>>>>>>>>>>>> opportunity to improve the overall security
> > > >> posture
> > > >>>>> when it
> > > >>>>>>>>> comes
> > > >>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> maintaining dependencies. The OWASP dependency
> > > >>> plugin
> > > >>>>>>> report
> > > >>>>>>>>>>>>>>> currently
> > > >>>>>>>>>>>>>>> generates 50 MB of HTML for questionable
> > > >>>> dependencies,
> > > >>>>> many
> > > >>>>>>>> of
> > > >>>>>>>>>>>>>>> which are
> > > >>>>>>>>>>>>>>> related to old versions of various libraries.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> As a starting point, here are a handful of
> > > >>> components
> > > >>>>> and
> > > >>>>>>>>>>>>>>> extension modules
> > > >>>>>>>>>>>>>>> that could be targeted for removal in a major
> > > >>>> version:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> - PostHTTP and GetHTTP
> > > >>>>>>>>>>>>>>> - ListenLumberjack and the entire
> > > >>>>> nifi-lumberjack-bundle
> > > >>>>>>>>>>>>>>> - ListenBeats and the entire nifi-beats-bundle
> > > >>>>>>>>>>>>>>> - Elasticsearch 5 components
> > > >>>>>>>>>>>>>>> - Hive 1 and 2 components
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> *Requiring Java 11*
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Java 8 is now over seven years old, and NiFi has
> > > >>>>> supported
> > > >>>>>>>>>> general
> > > >>>>>>>>>>>>>>> compatibility with Java 11 for several years.
> > > >> NiFi
> > > >>>>> 1.14.0
> > > >>>>>>>>>>>>>>> incorporated
> > > >>>>>>>>>>>>>>> internal improvements specifically related to TLS
> > > >>>> 1.3,
> > > >>>>>>> which
> > > >>>>>>>>>>>>>>> allowed
> > > >>>>>>>>>>>>>>> closing out the long-running Java 11
> > > >> compatibility
> > > >>>> epic
> > > >>>>>>>>>> NIFI-5174.
> > > >>>>>>>>>>>>>>> Making
> > > >>>>>>>>>>>>>>> Java 11 the minimum required version provides the
> > > >>>>>>> opportunity
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>>>> address
> > > >>>>>>>>>>>>>>> any lingering edge cases and put NiFi in a better
> > > >>>>> position
> > > >>>>>>> to
> > > >>>>>>>>>>>>>>> support
> > > >>>>>>>>>>>>>>> current Java versions.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> *JSR 310 for Date and Time Handling*
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Without making the scope too broad, transitioning
> > > >>>>> internal
> > > >>>>>>>> date
> > > >>>>>>>>>>>>>>> and time
> > > >>>>>>>>>>>>>>> handling to use DateTimeFormatter instead of
> > > >>>>>>> SimpleDateFormat
> > > >>>>>>>>>>>>>>> would provide
> > > >>>>>>>>>>>>>>> a number of advantages. The Java Time components
> > > >>>>> provide
> > > >>>>>>> much
> > > >>>>>>>>>>>>>>> better
> > > >>>>>>>>>>>>>>> clarity when it comes to handling localized date
> > > >>> and
> > > >>>>> time
> > > >>>>>>>>>>>>>>> representations,
> > > >>>>>>>>>>>>>>> and also avoid the inherent confusion of
> > > >>>> java.sql.Date
> > > >>>>>>>>> extending
> > > >>>>>>>>>>>>>>> java.util.Date. Many internal components,
> > > >>>> specifically
> > > >>>>>>>>>>>>>>> Record-oriented
> > > >>>>>>>>>>>>>>> processors and services, rely on date parsing,
> > > >>>> leading
> > > >>>>> to
> > > >>>>>>>>>>>>>>> confusion and
> > > >>>>>>>>>>>>>>> various workarounds. The pattern formats of
> > > >>>>>>> SimpleDateFormat
> > > >>>>>>>>> and
> > > >>>>>>>>>>>>>>> DateTimeFormatter are very similar, but there
> > > >> are a
> > > >>>> few
> > > >>>>>>>> subtle
> > > >>>>>>>>>>>>>>> differences.
> > > >>>>>>>>>>>>>>> Making this transition would provide a much
> > > >> better
> > > >>>>>>> foundation
> > > >>>>>>>>>>>>>>> going forward.
> > > >>>>>>>>>>>>>>> *Conclusion*
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks for giving this proposal some
> > > >> consideration.
> > > >>>>> Many of
> > > >>>>>>>> you
> > > >>>>>>>>>>>>>>> have been
> > > >>>>>>>>>>>>>>> developing NiFi for years and I look forward to
> > > >>> your
> > > >>>>>>>> feedback.
> > > >>>>>>>>> I
> > > >>>>>>>>>>>>>>> would be
> > > >>>>>>>>>>>>>>> glad to put together a more formalized
> > > >>> recommendation
> > > >>>>> on
> > > >>>>>>>>>>>>>>> Confluence and
> > > >>>>>>>>>>>>>>> write up Jira epics if this general approach
> > > >> sounds
> > > >>>>>>> agreeable
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> community.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>>>>> David Handermann
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Otto Fowler <ot...@gmail.com>.
Ryan, these are awesome, are there jiras for them? It would be best to
capture the requirements / use cases
From: Ryan Hendrickson <ry...@gmail.com>
<ry...@gmail.com>
Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
Date: August 4, 2021 at 01:14:23
To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
I'll preface with what I'm about to say is not rooted in any real
understanding of the lift required or other customer experiences... We
have used NiFi for a long time and are excited for what new features may
lay ahead in the 2.x line.
---After having written this, I think our succinct pain points focus
around large data flow management, auto-scaling container environments, and
integration of external analytic/enrichment creation.---
Thanks,
Ryan
1 - Integration with auto-scaling container environments
We often struggle with scaling custom processors in large flows. We've
implemented a recent design pattern where we write custom python code
packaged into a docker image, fronted with a web-service, and deployed into
a container environment. In NiFi we use InvokeHTTP to call out to the
web-service. If the load becomes too great, we can easily spin-up more
than one external web-service without sacrificing the NiFi's overall
thruput and thread contention. In this scenario, we're using NiFi to
manage the overall dataflow, routes, ETL services, etc, but not so much
smarts around auto-scaling processing, etc.
This makes it really easy to receive a dataflow analytic/enrichment from an
external team, wrap it in a docker image, and deploy it into an
auto-scaling container environment. This also lets us avoid conversations
about how performant the code is and if it's meeting a minimum throughput
requirement - typically XXGB/5minutes.
The request for 2.x would be to make interacting with container deployed
services a more seamless, streamlined process, possibly through a
normalized API or as a "Remote Container Environment" that can bind to a
service with a health endpoint and verify it's alive and well, etc.
2 - Integration of S3 buckets with Remote Process Groups for outage events
and data overruns
Another frequent struggle is the occasional death of a server in a NiFi
flow. Example -- Our overall NiFi flow consists of ~15 stand-alone NiFis
and 2 NiFi clusters. The stand-alone NiFi's are linearly chained together
with Remote Process Groups to perform the overall dataflow goals. If any
one of the 15 experience an outage, data begins to backup on the previous
NiFi, hitting backpressure limits, etc. When the down server comes back
online, it receives a flood of data, often overwhelming it, necessitating
careful flowfile limits and backpressure points.
The design pattern we're exploring, but would like to see baked into NiFi,
is that when a Remote Process Group becomes unavailable due to
server-to-server comms failures AND the backpressure limit is reached on a
relationship, the Remote Process Group would begin sending data to an S3
bucket as an automatic failover procedure. When the downed server comes
back-online, the Remote Process Group port would know to pull data from S3
AND receive data via the port from the chained server.
3 - Large Canvas Management
Beginning with a NiFi flow that consists of ~15 stand-alone NiFis and 2
NiFi clusters, each canvas is managed individually. To see the entire
canvas, systems engineering diagrams need to be kept up-to-date identifying
all the input/output ports. To move processors from server to server often
requires the verification of the existence of scripts, configuration files,
NiFi custom nars, and using the templating or Registry to migrate portions
of the dataflow.
Our current design pattern to accomplish some of these tasks include using
Ansible, Jenkins, and Nexus to verify script & configuration files are up
to date. We use Confluence's Gliffy to document the NiFi communication
paths between the servers.
Ideally, we'd be able to see an overall management view of the Canvases
together in a single place - it would indicate which server the portions of
the canvas are deployed on and tie together where the Remote Process Group
ports are connected. Additionally, NiFi would allow the user to move
Processors and Process Groups seamlessly from one server to another server
to help manually distribute and balance data flow processing cycles.
On Tue, Aug 3, 2021 at 12:45 PM Joe Witt <jo...@gmail.com> wrote:
> Kevin,
>
> This may well be a very interesting approach which opens up a lot of
> options for us in how we tackle a 2.x...
>
> Thanks
>
> On Tue, Aug 3, 2021 at 8:45 AM Kevin Doran <kd...@gmail.com>
> wrote:
> >
> > Outside of core NiFi, for MiNiFi, I think it would be beneficial to
> align MiNiFi Java and MiNiFI C++ to use the same method(s) for command
and
> control / flow deployment. This was discussed when MiNiFi Java was merged
> into the NiFi codebase [1].
> >
> > My proposal would be to create a Java implementation of the C2 Protocol
> used by MiNiFi C++, both server-side and client-side, to allow deploying
to
> MiNiFi Java instances from a centralize flow definition. Thanks to the
> unification of MiNiFi Java and NiFi, this could likely also be added as a
> capability of NiFi at that same time.
> >
> > If we are able to do this on the 1.x line, then I would also support
> deprecating other methods of flow deployment to MiNiFi outlined in [1]
and
> removing them in a 2.x release.
> >
> > [1] https://github.com/apache/nifi/pull/4933#issuecomment-808411977 <
> https://github.com/apache/nifi/pull/4933#issuecomment-808411977>
> > [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design <
> https://cwiki.apache.org/confluence/display/MINIFI/C2+Design>
> >
> > Thanks,
> > Kevin
> >
> > > On Aug 3, 2021, at 10:07 AM, David Handermann <
> exceptionfactory@gmail.com> wrote:
> > >
> > > Thanks for the continued discussion around what can or should be
> removed.
> > > Having a clear migration path from version 1 to version 2 will be
very
> > > important for supporting current deployments.
> > >
> > > Following the example of some other projects, one way to consider the
> > > upgrade is from the point of view of deprecation warnings. If
> implemented
> > > correctly, an installation of NiFi running the latest minor release
of
> > > version 1, and showing no deprecation warnings, should be upgradeable
> to
> > > version 2 without any changes. Making this work involves accurately
> tagging
> > > components and methods with deprecation indicators, and providing a
> clear
> > > way to observe deprecation warnings. With the current version
> containing
> > > both deprecated components and deprecated methods in various classes,
> this
> > > would involve some work to get right. A simple approach might be a
> named
> > > logger for deprecation warnings, which would write a separate log
> file. A
> > > more advanced approach might involve a tool to analyze the system and
> flow
> > > configurations. Either way, it seems that some additional work in a
> minor
> > > release version is necessary before considering a major release
> version.
> > >
> > > One other point on compatibility: as long as the core component API
> > > definition remains backwards compatible, it should be possible to run
> older
> > > components. This provides a transition option for customers using
> > > components such as PostHTTP, or any others that are not actively
> > > maintained. At some point it may become necessary to break
> compatibility at
> > > that level, but at least for version 2, maintaining component API
> > > compatibility is key.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Sun, Aug 1, 2021 at 10:23 AM Mark Bean <ma...@gmail.com>
> wrote:
> > >
> > >> I created a JIRA ticket to investigate and improve the performance
of
> > >> InvokeHTTP. It includes a flow definition for benchmarking the
> performance
> > >> of both PostHTTP and InvokeHTTP.
> > >>
> > >> https://issues.apache.org/jira/browse/NIFI-8968
> > >>
> > >> Thanks,
> > >> Mark
> > >>
> > >> On Fri, Jul 30, 2021 at 6:41 PM Adam Taft <ad...@adamtaft.com> wrote:
> > >>
> > >>> I'm not seeing the side thread that was going to discuss
deprecation
> of
> > >>> PostHTTP. Has that thread started and I just don't see it?
> > >>>
> > >>> One (significant?) concern with PostHTTP is the smooth integration
of
> > >>> NiFi-to-NiFi communication that is very transparently enabled with
> the
> > >>> ListenHTTP and PostHTTP processors. There's some special logic in
> there
> > >> for
> > >>> handling flowfiles that InvokeHTTP doesn't really (nor should
really)
> > >> have.
> > >>>
> > >>> I know of several (large) NiFi installations that rely on the
> PostHTTP /
> > >>> ListenHTTP combination. It has enabled NiFi to NiFi communication
for
> > >> folks
> > >>> reluctant or unable to enable site-to-site type configuration.
> > >>>
> > >>> Honestly, I don't know why we'd want to "deprecate" any processor,
as
> > >>> opposed to just moving it to a new location. If these processors
can
> be
> > >>> ported and maintained to whatever the 2.0 API looks like, there's
> > >> possibly
> > >>> little harm keeping them around.
> > >>>
> > >>> And by the way, what happened to the "marketplace" concept? Is this
> being
> > >>> considered for 2.0 as well? Because relocating the deprecated
> processors
> > >>> to an external nar might be the best solution. Losing PostHTTP
> entirely I
> > >>> think would be a mistake, but I'd conceptually support relocating
it.
> > >>>
> > >>> Thanks,
> > >>>
> > >>> /Adam
> > >>>
> > >>> On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <jo...@gmail.com>
wrote:
> > >>>
> > >>>> Looks like we just need to knock out 5 JIRAs :) [1]
> > >>>>
> > >>>> I felt like we had a label folks were using at one point but
quickly
> > >>>> looking revealed nothing exciting. After this confluence page
> > >>>> stabilizes a bit we can probably knock out some JIRAs and such.
> > >>>>
> > >>>> [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
> > >>>>
> > >>>> On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <
> ottobackwards@gmail.com>
> > >>>> wrote:
> > >>>>>
> > >>>>> I find myself wishing I had a list of all the jiras / issues that
> > >> have
> > >>>>> been put off for a 2.0 release because they required some change
or
> > >>>> another
> > >>>>> :(
> > >>>>>
> > >>>>> From: Joe Witt <jo...@gmail.com> <jo...@gmail.com>
> > >>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > >> dev@nifi.apache.org>
> > >>>>> Date: July 27, 2021 at 12:30:35
> > >>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <dev@nifi.apache.org
> >
> > >>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > >>>>>
> > >>>>> A few thoughts:
> > >>>>>
> > >>>>> 1. I would love to see deprecation notices show up in the UI in
> > >>>>> various ways to help motivate users to move off things to more
> > >>>>> supportable things. That is not a prerequisite for anything
> happening
> > >>>>> however. Just a good feature/nice thing to do for users when
> someone
> > >>>>> is able to tackle it.
> > >>>>>
> > >>>>> 2. The decision to deprecate something and to further remove it
> need
> > >>>>> not mean there is a superior solution available. If that thing
> itself
> > >>>>> isn't getting the love/attention it needs to be
> > >>>>> maintained/supported/healthy going forward that alone is enough
to
> > >>>>> remove it. That might well be the case with PostHTTP [1] and for
> > >>>>> comparison you can see how much effort has gone into InvokeHTTP
> [2].
> > >>>>>
> > >>>>> 3. When discussing a 2.0 release each thing we add as a 'must do'
> the
> > >>>>> further away from reality such a release will become. We'll have
to
> > >>>>> get very specific about 'musts' vs 'wants'.
> > >>>>>
> > >>>>> [1]
> > >>>>>
> > >>>>
> > >>>
> > >>
>
https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
> > >>>>> [2]
> > >>>>>
> > >>>>
> > >>>
> > >>
>
https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
> > >>>>>
> > >>>>> On Tue, Jul 27, 2021 at 8:47 AM David Handermann
> > >>>>> <ex...@apache.org> wrote:
> > >>>>>>
> > >>>>>> Thanks Mark, providing a template or comparison statistics with
> > >> Java
> > >>>>>> versions and component configuration details would be very
> helpful.
> > >>> If
> > >>>> it
> > >>>>>> is possible to run tests using a public API or deployable
service,
> > >>> that
> > >>>>>> would also help confirm potential differences.
> > >>>>>>
> > >>>>>> Showing a deprecation notice in the UI could be helpful, perhaps
> > >> as a
> > >>>>>> configurable option. NIFI-8650 describes a general Flow Analysis
> > >>>>>> capability, and it seems like that might be one possible way to
> > >>> surface
> > >>>>>> deprecation warnings. For something more specific to component
> > >>>>> deprecation,
> > >>>>>> it seems important to find a balance between making it obvious
and
> > >>>> making
> > >>>>>> it something that ends up getting ignored.
> > >>>>>>
> > >>>>>> Regards,
> > >>>>>> David Handermann
> > >>>>>>
> > >>>>>> On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <mark.o.bean@gmail.com
> >
> > >>>> wrote:
> > >>>>>>
> > >>>>>>> I'll start a new thread for PostHTTP when I get a template
and/or
> > >>>>> detailed
> > >>>>>>> stats.
> > >>>>>>>
> > >>>>>>> I know the deprecation is noted in the documentation. That's a
> > >>>>> necessary
> > >>>>>>> and minimum level of notification. I was suggesting it be more
> > >>>> obvious
> > >>>>> in
> > >>>>>>> the UI. I think it would be beneficial to somehow be aware of
the
> > >>>>>>> deprecation status simply by looking at the flow (perhaps even
on
> > >>> the
> > >>>>>>> summary pages too), and without having to open the
documentation
> > >>> for
> > >>>>> every
> > >>>>>>> processor to confirm whether or not a component is marked as
> > >>>>> deprecated.
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Mark
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
> > >>>>>>> exceptionfactory@apache.org> wrote:
> > >>>>>>>
> > >>>>>>>> Mark,
> > >>>>>>>>
> > >>>>>>>> Thanks for the feedback. It may be better to start a separate
> > >>>> thread
> > >>>>> on
> > >>>>>>>> PostHTTP, but can you provide an example flow demonstrating
the
> > >>>>>>> performance
> > >>>>>>>> differences between PostHTTP and InvokeHTTP?
> > >>>>>>>>
> > >>>>>>>> PostHTTP uses the Apache HttpComponents library, whereas
> > >>> InvokeHTTP
> > >>>>> uses
> > >>>>>>>> OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates
> > >> for
> > >>>>> OkHttp,
> > >>>>>>>> so it would be important to test with the most recent version.
> > >> It
> > >>>> is
> > >>>>> also
> > >>>>>>>> worth noting that test classes for PostHTTP are now
integration
> > >>>> tests
> > >>>>> as
> > >>>>>>>> opposed to unit tests, which means they are not executed as
> > >> part
> > >>> of
> > >>>>> the
> > >>>>>>>> automated builds.
> > >>>>>>>>
> > >>>>>>>> The NiFi documentation includes the contents of the
> > >>>> DeprecationNotice
> > >>>>> for
> > >>>>>>>> PostHTTP:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
>
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
> > >>>>>>>>
> > >>>>>>>> Regards,
> > >>>>>>>> David Handermann
> > >>>>>>>>
> > >>>>>>>> On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <
> > >> mark.o.bean@gmail.com
> > >>>>
> > >>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> I'm strongly in favor of reducing tech debt, and moving
> > >>>>> deliberately
> > >>>>>>> to a
> > >>>>>>>>> 2.0 release. I have a concern with just one processor that is
> > >>>>> currently
> > >>>>>>>>> marked as deprecated: PostHTTP. (I have not evaluated
> > >>>> specifically
> > >>>>> any
> > >>>>>>>>> other deprecated components; I cannot say if there are or are
> > >>> not
> > >>>>>>> similar
> > >>>>>>>>> issues elsewhere.) I understand the rationale for deprecating
> > >>>> this
> > >>>>>>>>> processor in that it eliminates a processor whose
> > >> functionality
> > >>>> is
> > >>>>>>>>> available elsewhere, specifically in the more flexible
> > >>>> InvokeHTTP.
> > >>>>>>>> However,
> > >>>>>>>>> in my experience and testing, PostHTTP performs transfers
> > >> with
> > >>>> far
> > >>>>>>>> greater
> > >>>>>>>>> throughput than InvokeHTTP. I would not be in favor of
> > >> removing
> > >>>>>>> PostHTTP
> > >>>>>>>>> unless/until InvokeHTTP is refactored to increase its
> > >>> throughput
> > >>>>>>>>> capability.
> > >>>>>>>>>
> > >>>>>>>>> Has anyone else continued to use PostHTTP over InvokeHTTP for
> > >>>>> similar
> > >>>>>>>>> reasons? Or, is there a performance-related configuration for
> > >>>>>>> InvokeHTTP
> > >>>>>>>> I
> > >>>>>>>>> may have missed?
> > >>>>>>>>>
> > >>>>>>>>> Also, in order to help facilitate a smooth transition to 2.0
> > >>>> from a
> > >>>>>>> user
> > >>>>>>>>> perspective, would it be advisable to add some sort of visual
> > >>>>> indicator
> > >>>>>>>> in
> > >>>>>>>>> the UI for components that are currently annotated as
> > >>>> @Deprecated?
> > >>>>> This
> > >>>>>>>>> might help users proactively modify their flow prior to a
> > >>> release
> > >>>>> that
> > >>>>>>>>> would otherwise break it.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <
> > >> bbende@gmail.com>
> > >>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> With the merging of MiNiFi and registry into the NiFi repo,
> > >>>> we've
> > >>>>>>>>>> actually gone more towards a mono-repo where everything is
> > >>>>> released
> > >>>>>>>>>> together. Eventually it would still be nice to have a
> > >> smaller
> > >>>>> base
> > >>>>>>>>>> distribution containing just the framework and standard
> > >> NARs,
> > >>>> but
> > >>>>> it
> > >>>>>>>>>> is hard to tackle that until we provide a way for users to
> > >>>> easily
> > >>>>>>>>>> obtain the NARs in some other way.
> > >>>>>>>>>>
> > >>>>>>>>>> On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <
> > >>>>> edward.armes@gmail.com
> > >>>>>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Given the major version number shift and the spliting up
> > >> of
> > >>>>>>>> processors
> > >>>>>>>>>> into
> > >>>>>>>>>>> multiple NAR's. I'd like to suggest that we start
> > >>>> individually
> > >>>>>>>>> versioning
> > >>>>>>>>>>> individual NARs/ bundles.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I can see this bringing a large number of benifits
> > >>> including
> > >>>>> making
> > >>>>>>>>> Nifi
> > >>>>>>>>>>> more deployable with things RPM, but also potentially
> > >>>> allowing
> > >>>>>>> those
> > >>>>>>>>> that
> > >>>>>>>>>>> have to deploy managed Nifi instances easier to upgrade.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Edward
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Mon, 26 Jul 2021, 20:42 Otto Fowler, <
> > >>>> ottobackwards@gmail.com>
> > >>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> The issue with updating the aws sdk is if it breaks any
> > >>> one
> > >>>>> of
> > >>>>>>> the
> > >>>>>>>>>>>> processors.
> > >>>>>>>>>>>> the Web Gateway API invoke processor for example is not
> > >>>> using
> > >>>>> a
> > >>>>>>>> high
> > >>>>>>>>>> level
> > >>>>>>>>>>>> purpose build client and may break.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> If we change the aws version, we need to coordinate in
> > >>>> such a
> > >>>>> way
> > >>>>>>>>> that
> > >>>>>>>>>> they
> > >>>>>>>>>>>> all
> > >>>>>>>>>>>> can come along reasonably.
> > >>>>>>>>>>>> IE: what happens if 1 or 2 break but the rest or OK?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> From: David Handermann <ex...@apache.org>
> > >>>>>>>>>>>> <ex...@apache.org>
> > >>>>>>>>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > >>>>>>>>> dev@nifi.apache.org>
> > >>>>>>>>>>>> Date: July 26, 2021 at 09:33:42
> > >>>>>>>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
> > >>>>>>> dev@nifi.apache.org
> > >>>>>>>>>
> > >>>>>>>>>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Chris,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for the reply and recommendations. It seems like
> > >>>> some
> > >>>>> of
> > >>>>>>> the
> > >>>>>>>>>> work to
> > >>>>>>>>>>>> reorganize the module structure could be done outside
> > >> of
> > >>> a
> > >>>>> major
> > >>>>>>>>>> release,
> > >>>>>>>>>>>> but it would be great to target any breaking changes
> > >> for
> > >>>> 2.0.
> > >>>>>>>>> Perhaps a
> > >>>>>>>>>>>> separate feature proposal on module restructuring, with
> > >>> the
> > >>>>> goal
> > >>>>>>> of
> > >>>>>>>>>>>> supporting optimized builds, would be a helpful way to
> > >>> move
> > >>>>> that
> > >>>>>>>> part
> > >>>>>>>>>> of
> > >>>>>>>>>>>> the discussion forward.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Regarding updating AWS SDK to version 2, it seems like
> > >>> that
> > >>>>> might
> > >>>>>>>> be
> > >>>>>>>>>>>> possible now. I haven't taken a close look at the
> > >>>> referencing
> > >>>>>>>>>> components,
> > >>>>>>>>>>>> so I'm not sure about the level of effort involved.
> > >> Minor
> > >>>>> NiFi
> > >>>>>>>>> version
> > >>>>>>>>>>>> updates have incorporated major new versions of
> > >>>> dependencies.
> > >>>>> For
> > >>>>>>>>>> example,
> > >>>>>>>>>>>> NiFi 1.14 included an upgrade from Spring Framework 4
> > >> to
> > >>> 5.
> > >>>>> On
> > >>>>>>> the
> > >>>>>>>>> one
> > >>>>>>>>>>>> hand, including the AWS SDK update as part of a major
> > >>>> release
> > >>>>>>> seems
> > >>>>>>>>>>>> helpful, but unless there are changes that break
> > >> existing
> > >>>>>>> component
> > >>>>>>>>>>>> properties, upgrading the AWS SDK could be worked
> > >>>>> independently.
> > >>>>>>>>>> Others may
> > >>>>>>>>>>>> have more insight into particular usage of that
> > >> library.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Regards,
> > >>>>>>>>>>>> David Handermann
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > >>>>>>>>>>>> <ch...@naimuri.com.invalid> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Might be worth considering refactoring the build as
> > >>> part
> > >>>> of
> > >>>>>>> this
> > >>>>>>>>> work
> > >>>>>>>>>>>> too,
> > >>>>>>>>>>>>> e.g. only building the bits of the repo affected by a
> > >>>>> commit,
> > >>>>>>>> etc.
> > >>>>>>>>> -
> > >>>>>>>>>>>>> discussed briefly in previous threads but don't think
> > >>> any
> > >>>>>>> changes
> > >>>>>>>>>> made
> > >>>>>>>>>>>> yet.
> > >>>>>>>>>>>>> If NARs/components are likely to be split up and
> > >>>> refactored
> > >>>>>>> then
> > >>>>>>>>> such
> > >>>>>>>>>>>> work
> > >>>>>>>>>>>>> around the build probably makes sense to consider.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I've a couple of PRs open that include updates to
> > >>>>> Elasticsearch
> > >>>>>>>>>> versions
> > >>>>>>>>>>>>> already, although I stopped at 7.10.2 (after which
> > >>>> Elastic
> > >>>>>>>> changed
> > >>>>>>>>>>>> licence)
> > >>>>>>>>>>>>> in case there were licence concerns. But more work
> > >> can
> > >>> be
> > >>>>> done
> > >>>>>>> to
> > >>>>>>>>>> tidy up
> > >>>>>>>>>>>>> the processors, absolutely.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> AWS libraries to v2 would seem a sensible move and a
> > >>>>> refactor
> > >>>>>>> of
> > >>>>>>>>>> those
> > >>>>>>>>>>>>> processors as well.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Chris Sampson
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > >>>>>>>>>>>> exceptionfactory@apache.org>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for pointing out the standard NAR bundles,
> > >>> Mark.
> > >>>>> There
> > >>>>>>>>> are a
> > >>>>>>>>>>>>> number
> > >>>>>>>>>>>>>> of components in the standard NAR bundles with
> > >>>> particular
> > >>>>>>>>>> dependencies
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>> would make more sense in separate NARs.
> > >> Reorganizing
> > >>>> the
> > >>>>>>>> standard
> > >>>>>>>>>> NAR
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>> components with limited dependencies and wide
> > >>>>> applicability
> > >>>>>>>> would
> > >>>>>>>>>>>>>> definitely help with future maintenance.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>>> David Handermann
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> > >>>>>>>>> markap14@hotmail.com>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> There’s also some code that exists in order to
> > >>>> maintain
> > >>>>>>>>> backward
> > >>>>>>>>>>>>>>> compatibility in the repositories. I would very
> > >>> much
> > >>>>> like
> > >>>>>>> the
> > >>>>>>>>>>>>>> repositories
> > >>>>>>>>>>>>>>> to contain no unnecessary code. And swap file
> > >>> format
> > >>>>>>> supports
> > >>>>>>>>>> really
> > >>>>>>>>>>>>> old
> > >>>>>>>>>>>>>>> formats. And the old impls of the repositories
> > >>>>> themselves,
> > >>>>>>>> like
> > >>>>>>>>>>>>>>> PersistentProvRepo instead of WriteAheadProv
> > >> Repo,
> > >>>> etc.
> > >>>>>>> Lots
> > >>>>>>>> of
> > >>>>>>>>>> stuff
> > >>>>>>>>>>>>>> there
> > >>>>>>>>>>>>>>> that can be removed. And some methods in
> > >>>> ProcessSession
> > >>>>>>> that
> > >>>>>>>>> are
> > >>>>>>>>>>>> never
> > >>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>> by any processor in the codebase but exists in
> > >> the
> > >>>>> public
> > >>>>>>> API
> > >>>>>>>>> so
> > >>>>>>>>>>>> can’t
> > >>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>> removed till 2.0.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I think his is also a great time to clean up the
> > >>>>> “standard
> > >>>>>>>>> nar.”
> > >>>>>>>>>> At
> > >>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>> point, it’s something like 70 MB. And many of the
> > >>>>>>> components
> > >>>>>>>>>> there
> > >>>>>>>>>>>> are
> > >>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>> really “standard” - things like connecting to
> > >> FTP &
> > >>>>> SFTP
> > >>>>>>>>>> servers, XML
> > >>>>>>>>>>>>>>> processing, Jolt transform, etc. could
> > >> potentially
> > >>> be
> > >>>>> moved
> > >>>>>>>>> into
> > >>>>>>>>>>>> other
> > >>>>>>>>>>>>>>> nars. The
> > >>>>> nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
> > >>>>>>> is
> > >>>>>>>>>> 6.9 MB
> > >>>>>>>>>>>> is
> > >>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>> necessary for stateless or minifi java. Lots of
> > >>>> things
> > >>>>>>>> probably
> > >>>>>>>>>> to
> > >>>>>>>>>>>>>>> reconsider within the standard nar.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I definitely think this is a reasonable approach,
> > >>> to
> > >>>>> allow
> > >>>>>>>> for
> > >>>>>>>>> a
> > >>>>>>>>>> 2.0
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>> is not a huge feature release but allows the
> > >>> project
> > >>>> to
> > >>>>> be
> > >>>>>>>>>> simpler
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>> nimble.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks
> > >>>>>>>>>>>>>>> -Mark
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > >>>>>>>>>> mikerthomsen@gmail.com
> > >>>>>>>>>>>>>> <mailto:
> > >>>>>>>>>>>>>>> mikerthomsen@gmail.com>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Russell,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> AFAICT from looking at Elastic's repos, the low
> > >>> level
> > >>>>> REST
> > >>>>>>>>>> client is
> > >>>>>>>>>>>>>>> still fine.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
>
https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Our Elasticsearch support is spread over two NARs
> > >>> at
> > >>>>>>> present.
> > >>>>>>>>> One
> > >>>>>>>>>>>> uses
> > >>>>>>>>>>>>>>> OkHttp and the other uses that low level Elastic
> > >>> REST
> > >>>>>>> client.
> > >>>>>>>>>>>>>>> Therefore, I think we're fine on licensing for
> > >> the
> > >>>>> moment.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Mike
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > >>>>>>>>>>>> russ@windofkeltia.com
> > >>>>>>>>>>>>>>> <ma...@windofkeltia.com>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Bringing up Elastic also reminds me that the
> > >>> Elastic
> > >>>>>>>> framework
> > >>>>>>>>>> has
> > >>>>>>>>>>>> just
> > >>>>>>>>>>>>>>> recently transitioned out of Open Source, so to
> > >>>>> acknowledge
> > >>>>>>>>> that,
> > >>>>>>>>>>>> maybe
> > >>>>>>>>>>>>>>> some effort toward OpenSearch--I say this not
> > >>>>> understanding
> > >>>>>>>>>> exactly
> > >>>>>>>>>>>> how
> > >>>>>>>>>>>>>>> this sort of thing is considered in a
> > >> large-scale,
> > >>>>>>>> world-class
> > >>>>>>>>>>>> software
> > >>>>>>>>>>>>>>> project like Apache NiFi. (I'm not a contributor,
> > >>>> just
> > >>>>> a
> > >>>>>>>>> grateful
> > >>>>>>>>>>>>>>> consumer.)
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Russ
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On 7/23/21 10:28 AM, Matt Burgess wrote:
> > >>>>>>>>>>>>>>> Along with the itemized list for ancient
> > >> components
> > >>>> we
> > >>>>>>> should
> > >>>>>>>>>> look at
> > >>>>>>>>>>>>>>> updating versions of drivers, SDKs, etc. for
> > >>> external
> > >>>>>>> systems
> > >>>>>>>>>> such as
> > >>>>>>>>>>>>>>> Elasticsearch, Cassandra, etc. There may be
> > >>> breaking
> > >>>>>>> changes
> > >>>>>>>>> but
> > >>>>>>>>>> 2.0
> > >>>>>>>>>>>>>>> is probably the right time to get things up to
> > >> date
> > >>>> to
> > >>>>> make
> > >>>>>>>>> them
> > >>>>>>>>>> more
> > >>>>>>>>>>>>>>> useful to more people.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > >>>>>>>>>> thenatog@gmail.com
> > >>>>>>>>>>>>>> <mailto:
> > >>>>>>>>>>>>>>> thenatog@gmail.com>> wrote:
> > >>>>>>>>>>>>>>> I'm a +1 for removing pretty much all of this
> > >>> stuff.
> > >>>>> There
> > >>>>>>>> are
> > >>>>>>>>>>>> security
> > >>>>>>>>>>>>>>> implications to keeping old dependencies around,
> > >> so
> > >>>> the
> > >>>>>>> more
> > >>>>>>>>> old
> > >>>>>>>>>> code
> > >>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>> can remove the better. I agree that eventually we
> > >>>> need
> > >>>>> to
> > >>>>>>>> move
> > >>>>>>>>> to
> > >>>>>>>>>>>>>>> supporting only Java 11+, and as our next release
> > >>>> will
> > >>>>>>>> probably
> > >>>>>>>>>> be
> > >>>>>>>>>>>>> about
> > >>>>>>>>>>>>>> 4
> > >>>>>>>>>>>>>>> - 6 months from now that doesn't seem too soon.
> > >> We
> > >>>>> could
> > >>>>>>>>>> potentially
> > >>>>>>>>>>>>>> break
> > >>>>>>>>>>>>>>> this in two and remove the deprecated processors
> > >>> and
> > >>>>> leave
> > >>>>>>>> 1.x
> > >>>>>>>>> on
> > >>>>>>>>>>>> Java
> > >>>>>>>>>>>>> 8,
> > >>>>>>>>>>>>>>> and finally start on 2.x which would support only
> > >>>> Java
> > >>>>> 11.
> > >>>>>>>> I'm
> > >>>>>>>>>> unsure
> > >>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>> what implications changing the date and time
> > >>> handling
> > >>>>> would
> > >>>>>>>>> have
> > >>>>>>>>>> -
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>>>>> running systems that use long term historical
> > >> logs,
> > >>>>>>>> unexpected
> > >>>>>>>>>>>> impacts
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> time logging could be a problem.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> As Joe says I think feature work will have to be
> > >>>>> dedicated
> > >>>>>>> to
> > >>>>>>>>>> 2.x and
> > >>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>> could support 1.x for security fixes for some
> > >>> period
> > >>>> of
> > >>>>>>> time.
> > >>>>>>>>> 2.x
> > >>>>>>>>>>>> seems
> > >>>>>>>>>>>>>>> like a gargantuan task but it's probably time to
> > >>> get
> > >>>>>>> started.
> > >>>>>>>>> Not
> > >>>>>>>>>>>> sure
> > >>>>>>>>>>>>>> how
> > >>>>>>>>>>>>>>> we handle all open PRs and the transition between
> > >>> 1.x
> > >>>>> and
> > >>>>>>>> 2.x.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> > >>>>>>>> joe.witt@gmail.com
> > >>>>>>>>>>>> <mailto:
> > >>>>>>>>>>>>>>> joe.witt@gmail.com>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Jon
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> You're right we have to be careful and you're
> > >> right
> > >>>>> there
> > >>>>>>> are
> > >>>>>>>>>> still
> > >>>>>>>>>>>>>>> significant Java 8 users out there. But we also
> > >>> have
> > >>>> to
> > >>>>> be
> > >>>>>>>>>> careful
> > >>>>>>>>>>>>>>> about security and sustainability of the
> > >> codebase.
> > >>> If
> > >>>>> we
> > >>>>>>> had
> > >>>>>>>>>> talked
> > >>>>>>>>>>>>>>> about this last year when that article came out
> > >> I'd
> > >>>>> have
> > >>>>>>>> agreed
> > >>>>>>>>>> it is
> > >>>>>>>>>>>>>>> too early. Interestingly that link seems to get
> > >>>> updated
> > >>>>>>> and I
> > >>>>>>>>>> tried
> > >>>>>>>>>>>>>>> [1] and found more recent data (not sure how
> > >>> recent).
> > >>>>>>> Anyway
> > >>>>>>>> it
> > >>>>>>>>>>>>>>> suggests Java 8 is still the top dog but we see
> > >>> good
> > >>>>> growth
> > >>>>>>>> on
> > >>>>>>>>>> 11. In
> > >>>>>>>>>>>>>>> my $dayjob this aligns to what I'm seeing too.
> > >>>>> Customers
> > >>>>>>>> didn't
> > >>>>>>>>>> seem
> > >>>>>>>>>>>>>>> to care about Java 11 until later half last year
> > >>> and
> > >>>>> now
> > >>>>>>>>>> suddenly it
> > >>>>>>>>>>>>>>> is all over the place.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I think once we put out a NiFi 2.0 release we'd
> > >> see
> > >>>>> rapid
> > >>>>>>>>>> decrease in
> > >>>>>>>>>>>>>>> work on the 1.x line just being blunt. We did
> > >> this
> > >>>> many
> > >>>>>>> years
> > >>>>>>>>> ago
> > >>>>>>>>>>>>>>> with 0.x to 1.x and we stood behind 0.x for a
> > >> while
> > >>>>> (maybe
> > >>>>>>> a
> > >>>>>>>>>> year or
> > >>>>>>>>>>>>>>> so) but it was purely bug fix/security related
> > >>> bits.
> > >>>> We
> > >>>>>>> would
> > >>>>>>>>>> need to
> > >>>>>>>>>>>>>>> do something similar. But feature work would
> > >> almost
> > >>>>>>> certainly
> > >>>>>>>>> go
> > >>>>>>>>>> to
> > >>>>>>>>>>>>>>> the 2.x line. Maybe there are other workable
> > >> models
> > >>>> but
> > >>>>> my
> > >>>>>>>>>> instinct
> > >>>>>>>>>>>>>>> suggests this is likely to follow a similar path.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> ...anyway I agree it isn't that easy of a call to
> > >>>> dump
> > >>>>> Java
> > >>>>>>>> 8.
> > >>>>>>>>> We
> > >>>>>>>>>>>>>>> need to make the call in both the interests of
> > >> the
> > >>>> user
> > >>>>>>> base
> > >>>>>>>>> and
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>> contributor base of the community.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> [1]
> > >>>> https://www.jetbrains.com/lp/devecosystem-2021/java/
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks
> > >>>>>>>>>>>>>>> Joe
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
> > >>>>>>> joe.witt@gmail.com
> > >>>>>>>>>> <mailto:
> > >>>>>>>>>>>>>>> joe.witt@gmail.com>> wrote:
> > >>>>>>>>>>>>>>> Russ
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Yeah the flow registry is a key part of it. But
> > >>> also
> > >>>>> now
> > >>>>>>> you
> > >>>>>>>>> can
> > >>>>>>>>>>>>>>> download the flow definition in JSON (upload i
> > >>> think
> > >>>> is
> > >>>>>>> there
> > >>>>>>>>> now
> > >>>>>>>>>>>>>>> too). Templates offered a series of challenges
> > >> such
> > >>>> as
> > >>>>> we
> > >>>>>>>> store
> > >>>>>>>>>> them
> > >>>>>>>>>>>>>>> in the flow definition which has made flows
> > >> massive
> > >>>> in
> > >>>>> an
> > >>>>>>>>>> unintended
> > >>>>>>>>>>>>>>> way which isn't fun for cluster behavior.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> We have a couple cases where we headed down a
> > >>>>> particular
> > >>>>>>>>> concept
> > >>>>>>>>>> and
> > >>>>>>>>>>>>>>> came up with better approaches later. We need to
> > >>>>> reconcile
> > >>>>>>>>> these
> > >>>>>>>>>> with
> > >>>>>>>>>>>>>>> the benefit of hindsight, and while being careful
> > >>> to
> > >>>> be
> > >>>>> not
> > >>>>>>>>>> overly
> > >>>>>>>>>>>>>>> disruptive to existing users, to reduce the
> > >>>>>>>>> codebase/maintenance
> > >>>>>>>>>>>>>>> burden and allow continued evolution of the
> > >>> project.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > >>>>>>>>>>>> russ@windofkeltia.com
> > >>>>>>>>>>>>>>> <ma...@windofkeltia.com>>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>> Joe,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I apologize for the off-topic intrusion, but what
> > >>>>> replaces
> > >>>>>>>>>> templates?
> > >>>>>>>>>>>>>>> The Registry? Templates rocked and we have used
> > >>> them
> > >>>>> since
> > >>>>>>>>> 0.5.x.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Russ
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On 7/23/21 8:31 AM, Joe Witt wrote:
> > >>>>>>>>>>>>>>> David,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I think this is a highly reasonable approach and
> > >>>> such a
> > >>>>>>> focus
> > >>>>>>>>>> will
> > >>>>>>>>>>>>>>> greatly help make a 2.0 release far more
> > >>> approachable
> > >>>>> to
> > >>>>>>>> knock
> > >>>>>>>>>> out.
> > >>>>>>>>>>>>>>> Not only that but tech debt reduction would help
> > >>> make
> > >>>>> work
> > >>>>>>>>>> towards
> > >>>>>>>>>>>>>>> major features we'd think about in a 'major
> > >>> release'
> > >>>>> sense
> > >>>>>>>> more
> > >>>>>>>>>>>>>>> approachable.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> We should remove all deprecated things (as well
> > >> as
> > >>>>> verify
> > >>>>>>> we
> > >>>>>>>>>> have the
> > >>>>>>>>>>>>>>> right list). We should remove/consider removal of
> > >>>>>>> deprecated
> > >>>>>>>>>>>>>>> concepts
> > >>>>>>>>>>>>>>> like templates. We should consider whether we can
> > >>>>> resolve
> > >>>>>>> the
> > >>>>>>>>>>>>>>> various
> > >>>>>>>>>>>>>>> ways we've handled what are now parameters down
> > >> to
> > >>>> one
> > >>>>>>> clean
> > >>>>>>>>>>>>>>> approach.
> > >>>>>>>>>>>>>>> We should remove options in the nifi.properties
> > >>> which
> > >>>>> turn
> > >>>>>>>> out
> > >>>>>>>>> to
> > >>>>>>>>>>>>>>> never be used quite right (if there are). There
> > >> is
> > >>>>> quite a
> > >>>>>>>> bit
> > >>>>>>>>> we
> > >>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>> do purely in the name of tech debt reduction.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Lots to consider here but I think this is the
> > >> right
> > >>>>>>>> discussion.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Than ks
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> > >>>>>>>> bbende@gmail.com
> > >>>>>>>>>>>> <mailto:
> > >>>>>>>>>>>>>>> bbende@gmail.com>>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>> I'm a +1 for this... Not sure if this falls under
> > >>>>> "Removing
> > >>>>>>>>>>>>>>> Deprecated
> > >>>>>>>>>>>>>>> Components", but I think we should also look at
> > >>>>> anything
> > >>>>>>> that
> > >>>>>>>>> has
> > >>>>>>>>>>>>>>> been
> > >>>>>>>>>>>>>>> marked as deprecated throughout the code base as
> > >> a
> > >>>>>>> candidate
> > >>>>>>>>> for
> > >>>>>>>>>>>>>>> removal. There are quite a few classes, methods,
> > >>>>>>> properties,
> > >>>>>>>>> etc
> > >>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>> have been waiting for a chance to be removed.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > >>>>>>>>>>>>>>> <exceptionfactory@apache.org<mailto:
> > >>>>>>>>> exceptionfactory@apache.org
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>> Team,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> With all of the excellent work that many have
> > >>>>> contributed
> > >>>>>>> to
> > >>>>>>>>> NiFi
> > >>>>>>>>>>>>>>> over the
> > >>>>>>>>>>>>>>> years, the code base has also accumulated some
> > >>> amount
> > >>>>> of
> > >>>>>>>>>> technical
> > >>>>>>>>>>>>>>> debt. A
> > >>>>>>>>>>>>>>> handful of components have been marked as
> > >>> deprecated,
> > >>>>> and
> > >>>>>>>> some
> > >>>>>>>>>>>>>>> components
> > >>>>>>>>>>>>>>> remain in the code base to support integration
> > >> with
> > >>>> old
> > >>>>>>>>> versions
> > >>>>>>>>>>>>>>> of various
> > >>>>>>>>>>>>>>> products. Following the principles of semantic
> > >>>>> versioning,
> > >>>>>>>>>>>>>>> introducing a
> > >>>>>>>>>>>>>>> major release would provide the opportunity to
> > >>> remove
> > >>>>> these
> > >>>>>>>>>>>>>>> deprecated and
> > >>>>>>>>>>>>>>> unsupported components.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Rather than focusing the next major release on
> > >> new
> > >>>>>>> features,
> > >>>>>>>>> what
> > >>>>>>>>>>>>>>> do you
> > >>>>>>>>>>>>>>> think about focusing on technical debt removal?
> > >>> This
> > >>>>>>> approach
> > >>>>>>>>>>>>>>> would not
> > >>>>>>>>>>>>>>> make for the most interesting release, but it
> > >>>> provides
> > >>>>> the
> > >>>>>>>>>>>>>>> opportunity to
> > >>>>>>>>>>>>>>> clean up elements that involve breaking changes.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Focusing on technical debt, at least three
> > >> primary
> > >>>>> goals
> > >>>>>>> come
> > >>>>>>>>> to
> > >>>>>>>>>>>>>>> mind for
> > >>>>>>>>>>>>>>> the next major release:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 1. Removal of deprecated and unmaintained
> > >>> components
> > >>>>>>>>>>>>>>> 2. Require Java 11 as the minimum supported
> > >> version
> > >>>>>>>>>>>>>>> 3. Transition internal date and time handling to
> > >>> JSR
> > >>>>> 310
> > >>>>>>>>>> java.time
> > >>>>>>>>>>>>>>> components
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> *Removing Deprecated Components*
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Removing support for older and deprecated
> > >>> components
> > >>>>>>>> provides a
> > >>>>>>>>>>>>>>> great
> > >>>>>>>>>>>>>>> opportunity to improve the overall security
> > >> posture
> > >>>>> when it
> > >>>>>>>>> comes
> > >>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> maintaining dependencies. The OWASP dependency
> > >>> plugin
> > >>>>>>> report
> > >>>>>>>>>>>>>>> currently
> > >>>>>>>>>>>>>>> generates 50 MB of HTML for questionable
> > >>>> dependencies,
> > >>>>> many
> > >>>>>>>> of
> > >>>>>>>>>>>>>>> which are
> > >>>>>>>>>>>>>>> related to old versions of various libraries.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> As a starting point, here are a handful of
> > >>> components
> > >>>>> and
> > >>>>>>>>>>>>>>> extension modules
> > >>>>>>>>>>>>>>> that could be targeted for removal in a major
> > >>>> version:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> - PostHTTP and GetHTTP
> > >>>>>>>>>>>>>>> - ListenLumberjack and the entire
> > >>>>> nifi-lumberjack-bundle
> > >>>>>>>>>>>>>>> - ListenBeats and the entire nifi-beats-bundle
> > >>>>>>>>>>>>>>> - Elasticsearch 5 components
> > >>>>>>>>>>>>>>> - Hive 1 and 2 components
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> *Requiring Java 11*
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Java 8 is now over seven years old, and NiFi has
> > >>>>> supported
> > >>>>>>>>>> general
> > >>>>>>>>>>>>>>> compatibility with Java 11 for several years.
> > >> NiFi
> > >>>>> 1.14.0
> > >>>>>>>>>>>>>>> incorporated
> > >>>>>>>>>>>>>>> internal improvements specifically related to TLS
> > >>>> 1.3,
> > >>>>>>> which
> > >>>>>>>>>>>>>>> allowed
> > >>>>>>>>>>>>>>> closing out the long-running Java 11
> > >> compatibility
> > >>>> epic
> > >>>>>>>>>> NIFI-5174.
> > >>>>>>>>>>>>>>> Making
> > >>>>>>>>>>>>>>> Java 11 the minimum required version provides the
> > >>>>>>> opportunity
> > >>>>>>>>> to
> > >>>>>>>>>>>>>>> address
> > >>>>>>>>>>>>>>> any lingering edge cases and put NiFi in a better
> > >>>>> position
> > >>>>>>> to
> > >>>>>>>>>>>>>>> support
> > >>>>>>>>>>>>>>> current Java versions.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> *JSR 310 for Date and Time Handling*
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Without making the scope too broad, transitioning
> > >>>>> internal
> > >>>>>>>> date
> > >>>>>>>>>>>>>>> and time
> > >>>>>>>>>>>>>>> handling to use DateTimeFormatter instead of
> > >>>>>>> SimpleDateFormat
> > >>>>>>>>>>>>>>> would provide
> > >>>>>>>>>>>>>>> a number of advantages. The Java Time components
> > >>>>> provide
> > >>>>>>> much
> > >>>>>>>>>>>>>>> better
> > >>>>>>>>>>>>>>> clarity when it comes to handling localized date
> > >>> and
> > >>>>> time
> > >>>>>>>>>>>>>>> representations,
> > >>>>>>>>>>>>>>> and also avoid the inherent confusion of
> > >>>> java.sql.Date
> > >>>>>>>>> extending
> > >>>>>>>>>>>>>>> java.util.Date. Many internal components,
> > >>>> specifically
> > >>>>>>>>>>>>>>> Record-oriented
> > >>>>>>>>>>>>>>> processors and services, rely on date parsing,
> > >>>> leading
> > >>>>> to
> > >>>>>>>>>>>>>>> confusion and
> > >>>>>>>>>>>>>>> various workarounds. The pattern formats of
> > >>>>>>> SimpleDateFormat
> > >>>>>>>>> and
> > >>>>>>>>>>>>>>> DateTimeFormatter are very similar, but there
> > >> are a
> > >>>> few
> > >>>>>>>> subtle
> > >>>>>>>>>>>>>>> differences.
> > >>>>>>>>>>>>>>> Making this transition would provide a much
> > >> better
> > >>>>>>> foundation
> > >>>>>>>>>>>>>>> going forward.
> > >>>>>>>>>>>>>>> *Conclusion*
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks for giving this proposal some
> > >> consideration.
> > >>>>> Many of
> > >>>>>>>> you
> > >>>>>>>>>>>>>>> have been
> > >>>>>>>>>>>>>>> developing NiFi for years and I look forward to
> > >>> your
> > >>>>>>>> feedback.
> > >>>>>>>>> I
> > >>>>>>>>>>>>>>> would be
> > >>>>>>>>>>>>>>> glad to put together a more formalized
> > >>> recommendation
> > >>>>> on
> > >>>>>>>>>>>>>>> Confluence and
> > >>>>>>>>>>>>>>> write up Jira epics if this general approach
> > >> sounds
> > >>>>>>> agreeable
> > >>>>>>>>> to
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> community.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>>>> David Handermann
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>
> > >>>
> > >>
> >
>
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Ryan Hendrickson <ry...@gmail.com>.
I'll preface with what I'm about to say is not rooted in any real
understanding of the lift required or other customer experiences... We
have used NiFi for a long time and are excited for what new features may
lay ahead in the 2.x line.
---After having written this, I think our succinct pain points focus
around large data flow management, auto-scaling container environments, and
integration of external analytic/enrichment creation.---
Thanks,
Ryan
1 - Integration with auto-scaling container environments
We often struggle with scaling custom processors in large flows. We've
implemented a recent design pattern where we write custom python code
packaged into a docker image, fronted with a web-service, and deployed into
a container environment. In NiFi we use InvokeHTTP to call out to the
web-service. If the load becomes too great, we can easily spin-up more
than one external web-service without sacrificing the NiFi's overall
thruput and thread contention. In this scenario, we're using NiFi to
manage the overall dataflow, routes, ETL services, etc, but not so much
smarts around auto-scaling processing, etc.
This makes it really easy to receive a dataflow analytic/enrichment from an
external team, wrap it in a docker image, and deploy it into an
auto-scaling container environment. This also lets us avoid conversations
about how performant the code is and if it's meeting a minimum throughput
requirement - typically XXGB/5minutes.
The request for 2.x would be to make interacting with container deployed
services a more seamless, streamlined process, possibly through a
normalized API or as a "Remote Container Environment" that can bind to a
service with a health endpoint and verify it's alive and well, etc.
2 - Integration of S3 buckets with Remote Process Groups for outage events
and data overruns
Another frequent struggle is the occasional death of a server in a NiFi
flow. Example -- Our overall NiFi flow consists of ~15 stand-alone NiFis
and 2 NiFi clusters. The stand-alone NiFi's are linearly chained together
with Remote Process Groups to perform the overall dataflow goals. If any
one of the 15 experience an outage, data begins to backup on the previous
NiFi, hitting backpressure limits, etc. When the down server comes back
online, it receives a flood of data, often overwhelming it, necessitating
careful flowfile limits and backpressure points.
The design pattern we're exploring, but would like to see baked into NiFi,
is that when a Remote Process Group becomes unavailable due to
server-to-server comms failures AND the backpressure limit is reached on a
relationship, the Remote Process Group would begin sending data to an S3
bucket as an automatic failover procedure. When the downed server comes
back-online, the Remote Process Group port would know to pull data from S3
AND receive data via the port from the chained server.
3 - Large Canvas Management
Beginning with a NiFi flow that consists of ~15 stand-alone NiFis and 2
NiFi clusters, each canvas is managed individually. To see the entire
canvas, systems engineering diagrams need to be kept up-to-date identifying
all the input/output ports. To move processors from server to server often
requires the verification of the existence of scripts, configuration files,
NiFi custom nars, and using the templating or Registry to migrate portions
of the dataflow.
Our current design pattern to accomplish some of these tasks include using
Ansible, Jenkins, and Nexus to verify script & configuration files are up
to date. We use Confluence's Gliffy to document the NiFi communication
paths between the servers.
Ideally, we'd be able to see an overall management view of the Canvases
together in a single place - it would indicate which server the portions of
the canvas are deployed on and tie together where the Remote Process Group
ports are connected. Additionally, NiFi would allow the user to move
Processors and Process Groups seamlessly from one server to another server
to help manually distribute and balance data flow processing cycles.
On Tue, Aug 3, 2021 at 12:45 PM Joe Witt <jo...@gmail.com> wrote:
> Kevin,
>
> This may well be a very interesting approach which opens up a lot of
> options for us in how we tackle a 2.x...
>
> Thanks
>
> On Tue, Aug 3, 2021 at 8:45 AM Kevin Doran <kd...@gmail.com>
> wrote:
> >
> > Outside of core NiFi, for MiNiFi, I think it would be beneficial to
> align MiNiFi Java and MiNiFI C++ to use the same method(s) for command and
> control / flow deployment. This was discussed when MiNiFi Java was merged
> into the NiFi codebase [1].
> >
> > My proposal would be to create a Java implementation of the C2 Protocol
> used by MiNiFi C++, both server-side and client-side, to allow deploying to
> MiNiFi Java instances from a centralize flow definition. Thanks to the
> unification of MiNiFi Java and NiFi, this could likely also be added as a
> capability of NiFi at that same time.
> >
> > If we are able to do this on the 1.x line, then I would also support
> deprecating other methods of flow deployment to MiNiFi outlined in [1] and
> removing them in a 2.x release.
> >
> > [1] https://github.com/apache/nifi/pull/4933#issuecomment-808411977 <
> https://github.com/apache/nifi/pull/4933#issuecomment-808411977>
> > [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design <
> https://cwiki.apache.org/confluence/display/MINIFI/C2+Design>
> >
> > Thanks,
> > Kevin
> >
> > > On Aug 3, 2021, at 10:07 AM, David Handermann <
> exceptionfactory@gmail.com> wrote:
> > >
> > > Thanks for the continued discussion around what can or should be
> removed.
> > > Having a clear migration path from version 1 to version 2 will be very
> > > important for supporting current deployments.
> > >
> > > Following the example of some other projects, one way to consider the
> > > upgrade is from the point of view of deprecation warnings. If
> implemented
> > > correctly, an installation of NiFi running the latest minor release of
> > > version 1, and showing no deprecation warnings, should be upgradeable
> to
> > > version 2 without any changes. Making this work involves accurately
> tagging
> > > components and methods with deprecation indicators, and providing a
> clear
> > > way to observe deprecation warnings. With the current version
> containing
> > > both deprecated components and deprecated methods in various classes,
> this
> > > would involve some work to get right. A simple approach might be a
> named
> > > logger for deprecation warnings, which would write a separate log
> file. A
> > > more advanced approach might involve a tool to analyze the system and
> flow
> > > configurations. Either way, it seems that some additional work in a
> minor
> > > release version is necessary before considering a major release
> version.
> > >
> > > One other point on compatibility: as long as the core component API
> > > definition remains backwards compatible, it should be possible to run
> older
> > > components. This provides a transition option for customers using
> > > components such as PostHTTP, or any others that are not actively
> > > maintained. At some point it may become necessary to break
> compatibility at
> > > that level, but at least for version 2, maintaining component API
> > > compatibility is key.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Sun, Aug 1, 2021 at 10:23 AM Mark Bean <ma...@gmail.com>
> wrote:
> > >
> > >> I created a JIRA ticket to investigate and improve the performance of
> > >> InvokeHTTP. It includes a flow definition for benchmarking the
> performance
> > >> of both PostHTTP and InvokeHTTP.
> > >>
> > >> https://issues.apache.org/jira/browse/NIFI-8968
> > >>
> > >> Thanks,
> > >> Mark
> > >>
> > >> On Fri, Jul 30, 2021 at 6:41 PM Adam Taft <ad...@adamtaft.com> wrote:
> > >>
> > >>> I'm not seeing the side thread that was going to discuss deprecation
> of
> > >>> PostHTTP. Has that thread started and I just don't see it?
> > >>>
> > >>> One (significant?) concern with PostHTTP is the smooth integration of
> > >>> NiFi-to-NiFi communication that is very transparently enabled with
> the
> > >>> ListenHTTP and PostHTTP processors. There's some special logic in
> there
> > >> for
> > >>> handling flowfiles that InvokeHTTP doesn't really (nor should really)
> > >> have.
> > >>>
> > >>> I know of several (large) NiFi installations that rely on the
> PostHTTP /
> > >>> ListenHTTP combination. It has enabled NiFi to NiFi communication for
> > >> folks
> > >>> reluctant or unable to enable site-to-site type configuration.
> > >>>
> > >>> Honestly, I don't know why we'd want to "deprecate" any processor, as
> > >>> opposed to just moving it to a new location. If these processors can
> be
> > >>> ported and maintained to whatever the 2.0 API looks like, there's
> > >> possibly
> > >>> little harm keeping them around.
> > >>>
> > >>> And by the way, what happened to the "marketplace" concept? Is this
> being
> > >>> considered for 2.0 as well? Because relocating the deprecated
> processors
> > >>> to an external nar might be the best solution. Losing PostHTTP
> entirely I
> > >>> think would be a mistake, but I'd conceptually support relocating it.
> > >>>
> > >>> Thanks,
> > >>>
> > >>> /Adam
> > >>>
> > >>> On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <jo...@gmail.com> wrote:
> > >>>
> > >>>> Looks like we just need to knock out 5 JIRAs :) [1]
> > >>>>
> > >>>> I felt like we had a label folks were using at one point but quickly
> > >>>> looking revealed nothing exciting. After this confluence page
> > >>>> stabilizes a bit we can probably knock out some JIRAs and such.
> > >>>>
> > >>>> [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
> > >>>>
> > >>>> On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <
> ottobackwards@gmail.com>
> > >>>> wrote:
> > >>>>>
> > >>>>> I find myself wishing I had a list of all the jiras / issues that
> > >> have
> > >>>>> been put off for a 2.0 release because they required some change or
> > >>>> another
> > >>>>> :(
> > >>>>>
> > >>>>> From: Joe Witt <jo...@gmail.com> <jo...@gmail.com>
> > >>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > >> dev@nifi.apache.org>
> > >>>>> Date: July 27, 2021 at 12:30:35
> > >>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <dev@nifi.apache.org
> >
> > >>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > >>>>>
> > >>>>> A few thoughts:
> > >>>>>
> > >>>>> 1. I would love to see deprecation notices show up in the UI in
> > >>>>> various ways to help motivate users to move off things to more
> > >>>>> supportable things. That is not a prerequisite for anything
> happening
> > >>>>> however. Just a good feature/nice thing to do for users when
> someone
> > >>>>> is able to tackle it.
> > >>>>>
> > >>>>> 2. The decision to deprecate something and to further remove it
> need
> > >>>>> not mean there is a superior solution available. If that thing
> itself
> > >>>>> isn't getting the love/attention it needs to be
> > >>>>> maintained/supported/healthy going forward that alone is enough to
> > >>>>> remove it. That might well be the case with PostHTTP [1] and for
> > >>>>> comparison you can see how much effort has gone into InvokeHTTP
> [2].
> > >>>>>
> > >>>>> 3. When discussing a 2.0 release each thing we add as a 'must do'
> the
> > >>>>> further away from reality such a release will become. We'll have to
> > >>>>> get very specific about 'musts' vs 'wants'.
> > >>>>>
> > >>>>> [1]
> > >>>>>
> > >>>>
> > >>>
> > >>
> https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
> > >>>>> [2]
> > >>>>>
> > >>>>
> > >>>
> > >>
> https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
> > >>>>>
> > >>>>> On Tue, Jul 27, 2021 at 8:47 AM David Handermann
> > >>>>> <ex...@apache.org> wrote:
> > >>>>>>
> > >>>>>> Thanks Mark, providing a template or comparison statistics with
> > >> Java
> > >>>>>> versions and component configuration details would be very
> helpful.
> > >>> If
> > >>>> it
> > >>>>>> is possible to run tests using a public API or deployable service,
> > >>> that
> > >>>>>> would also help confirm potential differences.
> > >>>>>>
> > >>>>>> Showing a deprecation notice in the UI could be helpful, perhaps
> > >> as a
> > >>>>>> configurable option. NIFI-8650 describes a general Flow Analysis
> > >>>>>> capability, and it seems like that might be one possible way to
> > >>> surface
> > >>>>>> deprecation warnings. For something more specific to component
> > >>>>> deprecation,
> > >>>>>> it seems important to find a balance between making it obvious and
> > >>>> making
> > >>>>>> it something that ends up getting ignored.
> > >>>>>>
> > >>>>>> Regards,
> > >>>>>> David Handermann
> > >>>>>>
> > >>>>>> On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <mark.o.bean@gmail.com
> >
> > >>>> wrote:
> > >>>>>>
> > >>>>>>> I'll start a new thread for PostHTTP when I get a template and/or
> > >>>>> detailed
> > >>>>>>> stats.
> > >>>>>>>
> > >>>>>>> I know the deprecation is noted in the documentation. That's a
> > >>>>> necessary
> > >>>>>>> and minimum level of notification. I was suggesting it be more
> > >>>> obvious
> > >>>>> in
> > >>>>>>> the UI. I think it would be beneficial to somehow be aware of the
> > >>>>>>> deprecation status simply by looking at the flow (perhaps even on
> > >>> the
> > >>>>>>> summary pages too), and without having to open the documentation
> > >>> for
> > >>>>> every
> > >>>>>>> processor to confirm whether or not a component is marked as
> > >>>>> deprecated.
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Mark
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
> > >>>>>>> exceptionfactory@apache.org> wrote:
> > >>>>>>>
> > >>>>>>>> Mark,
> > >>>>>>>>
> > >>>>>>>> Thanks for the feedback. It may be better to start a separate
> > >>>> thread
> > >>>>> on
> > >>>>>>>> PostHTTP, but can you provide an example flow demonstrating the
> > >>>>>>> performance
> > >>>>>>>> differences between PostHTTP and InvokeHTTP?
> > >>>>>>>>
> > >>>>>>>> PostHTTP uses the Apache HttpComponents library, whereas
> > >>> InvokeHTTP
> > >>>>> uses
> > >>>>>>>> OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates
> > >> for
> > >>>>> OkHttp,
> > >>>>>>>> so it would be important to test with the most recent version.
> > >> It
> > >>>> is
> > >>>>> also
> > >>>>>>>> worth noting that test classes for PostHTTP are now integration
> > >>>> tests
> > >>>>> as
> > >>>>>>>> opposed to unit tests, which means they are not executed as
> > >> part
> > >>> of
> > >>>>> the
> > >>>>>>>> automated builds.
> > >>>>>>>>
> > >>>>>>>> The NiFi documentation includes the contents of the
> > >>>> DeprecationNotice
> > >>>>> for
> > >>>>>>>> PostHTTP:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
> > >>>>>>>>
> > >>>>>>>> Regards,
> > >>>>>>>> David Handermann
> > >>>>>>>>
> > >>>>>>>> On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <
> > >> mark.o.bean@gmail.com
> > >>>>
> > >>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> I'm strongly in favor of reducing tech debt, and moving
> > >>>>> deliberately
> > >>>>>>> to a
> > >>>>>>>>> 2.0 release. I have a concern with just one processor that is
> > >>>>> currently
> > >>>>>>>>> marked as deprecated: PostHTTP. (I have not evaluated
> > >>>> specifically
> > >>>>> any
> > >>>>>>>>> other deprecated components; I cannot say if there are or are
> > >>> not
> > >>>>>>> similar
> > >>>>>>>>> issues elsewhere.) I understand the rationale for deprecating
> > >>>> this
> > >>>>>>>>> processor in that it eliminates a processor whose
> > >> functionality
> > >>>> is
> > >>>>>>>>> available elsewhere, specifically in the more flexible
> > >>>> InvokeHTTP.
> > >>>>>>>> However,
> > >>>>>>>>> in my experience and testing, PostHTTP performs transfers
> > >> with
> > >>>> far
> > >>>>>>>> greater
> > >>>>>>>>> throughput than InvokeHTTP. I would not be in favor of
> > >> removing
> > >>>>>>> PostHTTP
> > >>>>>>>>> unless/until InvokeHTTP is refactored to increase its
> > >>> throughput
> > >>>>>>>>> capability.
> > >>>>>>>>>
> > >>>>>>>>> Has anyone else continued to use PostHTTP over InvokeHTTP for
> > >>>>> similar
> > >>>>>>>>> reasons? Or, is there a performance-related configuration for
> > >>>>>>> InvokeHTTP
> > >>>>>>>> I
> > >>>>>>>>> may have missed?
> > >>>>>>>>>
> > >>>>>>>>> Also, in order to help facilitate a smooth transition to 2.0
> > >>>> from a
> > >>>>>>> user
> > >>>>>>>>> perspective, would it be advisable to add some sort of visual
> > >>>>> indicator
> > >>>>>>>> in
> > >>>>>>>>> the UI for components that are currently annotated as
> > >>>> @Deprecated?
> > >>>>> This
> > >>>>>>>>> might help users proactively modify their flow prior to a
> > >>> release
> > >>>>> that
> > >>>>>>>>> would otherwise break it.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <
> > >> bbende@gmail.com>
> > >>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> With the merging of MiNiFi and registry into the NiFi repo,
> > >>>> we've
> > >>>>>>>>>> actually gone more towards a mono-repo where everything is
> > >>>>> released
> > >>>>>>>>>> together. Eventually it would still be nice to have a
> > >> smaller
> > >>>>> base
> > >>>>>>>>>> distribution containing just the framework and standard
> > >> NARs,
> > >>>> but
> > >>>>> it
> > >>>>>>>>>> is hard to tackle that until we provide a way for users to
> > >>>> easily
> > >>>>>>>>>> obtain the NARs in some other way.
> > >>>>>>>>>>
> > >>>>>>>>>> On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <
> > >>>>> edward.armes@gmail.com
> > >>>>>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Given the major version number shift and the spliting up
> > >> of
> > >>>>>>>> processors
> > >>>>>>>>>> into
> > >>>>>>>>>>> multiple NAR's. I'd like to suggest that we start
> > >>>> individually
> > >>>>>>>>> versioning
> > >>>>>>>>>>> individual NARs/ bundles.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I can see this bringing a large number of benifits
> > >>> including
> > >>>>> making
> > >>>>>>>>> Nifi
> > >>>>>>>>>>> more deployable with things RPM, but also potentially
> > >>>> allowing
> > >>>>>>> those
> > >>>>>>>>> that
> > >>>>>>>>>>> have to deploy managed Nifi instances easier to upgrade.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Edward
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Mon, 26 Jul 2021, 20:42 Otto Fowler, <
> > >>>> ottobackwards@gmail.com>
> > >>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> The issue with updating the aws sdk is if it breaks any
> > >>> one
> > >>>>> of
> > >>>>>>> the
> > >>>>>>>>>>>> processors.
> > >>>>>>>>>>>> the Web Gateway API invoke processor for example is not
> > >>>> using
> > >>>>> a
> > >>>>>>>> high
> > >>>>>>>>>> level
> > >>>>>>>>>>>> purpose build client and may break.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> If we change the aws version, we need to coordinate in
> > >>>> such a
> > >>>>> way
> > >>>>>>>>> that
> > >>>>>>>>>> they
> > >>>>>>>>>>>> all
> > >>>>>>>>>>>> can come along reasonably.
> > >>>>>>>>>>>> IE: what happens if 1 or 2 break but the rest or OK?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> From: David Handermann <ex...@apache.org>
> > >>>>>>>>>>>> <ex...@apache.org>
> > >>>>>>>>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > >>>>>>>>> dev@nifi.apache.org>
> > >>>>>>>>>>>> Date: July 26, 2021 at 09:33:42
> > >>>>>>>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
> > >>>>>>> dev@nifi.apache.org
> > >>>>>>>>>
> > >>>>>>>>>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Chris,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for the reply and recommendations. It seems like
> > >>>> some
> > >>>>> of
> > >>>>>>> the
> > >>>>>>>>>> work to
> > >>>>>>>>>>>> reorganize the module structure could be done outside
> > >> of
> > >>> a
> > >>>>> major
> > >>>>>>>>>> release,
> > >>>>>>>>>>>> but it would be great to target any breaking changes
> > >> for
> > >>>> 2.0.
> > >>>>>>>>> Perhaps a
> > >>>>>>>>>>>> separate feature proposal on module restructuring, with
> > >>> the
> > >>>>> goal
> > >>>>>>> of
> > >>>>>>>>>>>> supporting optimized builds, would be a helpful way to
> > >>> move
> > >>>>> that
> > >>>>>>>> part
> > >>>>>>>>>> of
> > >>>>>>>>>>>> the discussion forward.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Regarding updating AWS SDK to version 2, it seems like
> > >>> that
> > >>>>> might
> > >>>>>>>> be
> > >>>>>>>>>>>> possible now. I haven't taken a close look at the
> > >>>> referencing
> > >>>>>>>>>> components,
> > >>>>>>>>>>>> so I'm not sure about the level of effort involved.
> > >> Minor
> > >>>>> NiFi
> > >>>>>>>>> version
> > >>>>>>>>>>>> updates have incorporated major new versions of
> > >>>> dependencies.
> > >>>>> For
> > >>>>>>>>>> example,
> > >>>>>>>>>>>> NiFi 1.14 included an upgrade from Spring Framework 4
> > >> to
> > >>> 5.
> > >>>>> On
> > >>>>>>> the
> > >>>>>>>>> one
> > >>>>>>>>>>>> hand, including the AWS SDK update as part of a major
> > >>>> release
> > >>>>>>> seems
> > >>>>>>>>>>>> helpful, but unless there are changes that break
> > >> existing
> > >>>>>>> component
> > >>>>>>>>>>>> properties, upgrading the AWS SDK could be worked
> > >>>>> independently.
> > >>>>>>>>>> Others may
> > >>>>>>>>>>>> have more insight into particular usage of that
> > >> library.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Regards,
> > >>>>>>>>>>>> David Handermann
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > >>>>>>>>>>>> <ch...@naimuri.com.invalid> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Might be worth considering refactoring the build as
> > >>> part
> > >>>> of
> > >>>>>>> this
> > >>>>>>>>> work
> > >>>>>>>>>>>> too,
> > >>>>>>>>>>>>> e.g. only building the bits of the repo affected by a
> > >>>>> commit,
> > >>>>>>>> etc.
> > >>>>>>>>> -
> > >>>>>>>>>>>>> discussed briefly in previous threads but don't think
> > >>> any
> > >>>>>>> changes
> > >>>>>>>>>> made
> > >>>>>>>>>>>> yet.
> > >>>>>>>>>>>>> If NARs/components are likely to be split up and
> > >>>> refactored
> > >>>>>>> then
> > >>>>>>>>> such
> > >>>>>>>>>>>> work
> > >>>>>>>>>>>>> around the build probably makes sense to consider.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I've a couple of PRs open that include updates to
> > >>>>> Elasticsearch
> > >>>>>>>>>> versions
> > >>>>>>>>>>>>> already, although I stopped at 7.10.2 (after which
> > >>>> Elastic
> > >>>>>>>> changed
> > >>>>>>>>>>>> licence)
> > >>>>>>>>>>>>> in case there were licence concerns. But more work
> > >> can
> > >>> be
> > >>>>> done
> > >>>>>>> to
> > >>>>>>>>>> tidy up
> > >>>>>>>>>>>>> the processors, absolutely.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> AWS libraries to v2 would seem a sensible move and a
> > >>>>> refactor
> > >>>>>>> of
> > >>>>>>>>>> those
> > >>>>>>>>>>>>> processors as well.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Chris Sampson
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > >>>>>>>>>>>> exceptionfactory@apache.org>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for pointing out the standard NAR bundles,
> > >>> Mark.
> > >>>>> There
> > >>>>>>>>> are a
> > >>>>>>>>>>>>> number
> > >>>>>>>>>>>>>> of components in the standard NAR bundles with
> > >>>> particular
> > >>>>>>>>>> dependencies
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>> would make more sense in separate NARs.
> > >> Reorganizing
> > >>>> the
> > >>>>>>>> standard
> > >>>>>>>>>> NAR
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>> components with limited dependencies and wide
> > >>>>> applicability
> > >>>>>>>> would
> > >>>>>>>>>>>>>> definitely help with future maintenance.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>>> David Handermann
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> > >>>>>>>>> markap14@hotmail.com>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> There’s also some code that exists in order to
> > >>>> maintain
> > >>>>>>>>> backward
> > >>>>>>>>>>>>>>> compatibility in the repositories. I would very
> > >>> much
> > >>>>> like
> > >>>>>>> the
> > >>>>>>>>>>>>>> repositories
> > >>>>>>>>>>>>>>> to contain no unnecessary code. And swap file
> > >>> format
> > >>>>>>> supports
> > >>>>>>>>>> really
> > >>>>>>>>>>>>> old
> > >>>>>>>>>>>>>>> formats. And the old impls of the repositories
> > >>>>> themselves,
> > >>>>>>>> like
> > >>>>>>>>>>>>>>> PersistentProvRepo instead of WriteAheadProv
> > >> Repo,
> > >>>> etc.
> > >>>>>>> Lots
> > >>>>>>>> of
> > >>>>>>>>>> stuff
> > >>>>>>>>>>>>>> there
> > >>>>>>>>>>>>>>> that can be removed. And some methods in
> > >>>> ProcessSession
> > >>>>>>> that
> > >>>>>>>>> are
> > >>>>>>>>>>>> never
> > >>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>> by any processor in the codebase but exists in
> > >> the
> > >>>>> public
> > >>>>>>> API
> > >>>>>>>>> so
> > >>>>>>>>>>>> can’t
> > >>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>> removed till 2.0.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I think his is also a great time to clean up the
> > >>>>> “standard
> > >>>>>>>>> nar.”
> > >>>>>>>>>> At
> > >>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>> point, it’s something like 70 MB. And many of the
> > >>>>>>> components
> > >>>>>>>>>> there
> > >>>>>>>>>>>> are
> > >>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>> really “standard” - things like connecting to
> > >> FTP &
> > >>>>> SFTP
> > >>>>>>>>>> servers, XML
> > >>>>>>>>>>>>>>> processing, Jolt transform, etc. could
> > >> potentially
> > >>> be
> > >>>>> moved
> > >>>>>>>>> into
> > >>>>>>>>>>>> other
> > >>>>>>>>>>>>>>> nars. The
> > >>>>> nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
> > >>>>>>> is
> > >>>>>>>>>> 6.9 MB
> > >>>>>>>>>>>> is
> > >>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>> necessary for stateless or minifi java. Lots of
> > >>>> things
> > >>>>>>>> probably
> > >>>>>>>>>> to
> > >>>>>>>>>>>>>>> reconsider within the standard nar.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I definitely think this is a reasonable approach,
> > >>> to
> > >>>>> allow
> > >>>>>>>> for
> > >>>>>>>>> a
> > >>>>>>>>>> 2.0
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>> is not a huge feature release but allows the
> > >>> project
> > >>>> to
> > >>>>> be
> > >>>>>>>>>> simpler
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>> nimble.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks
> > >>>>>>>>>>>>>>> -Mark
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > >>>>>>>>>> mikerthomsen@gmail.com
> > >>>>>>>>>>>>>> <mailto:
> > >>>>>>>>>>>>>>> mikerthomsen@gmail.com>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Russell,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> AFAICT from looking at Elastic's repos, the low
> > >>> level
> > >>>>> REST
> > >>>>>>>>>> client is
> > >>>>>>>>>>>>>>> still fine.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Our Elasticsearch support is spread over two NARs
> > >>> at
> > >>>>>>> present.
> > >>>>>>>>> One
> > >>>>>>>>>>>> uses
> > >>>>>>>>>>>>>>> OkHttp and the other uses that low level Elastic
> > >>> REST
> > >>>>>>> client.
> > >>>>>>>>>>>>>>> Therefore, I think we're fine on licensing for
> > >> the
> > >>>>> moment.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Mike
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > >>>>>>>>>>>> russ@windofkeltia.com
> > >>>>>>>>>>>>>>> <ma...@windofkeltia.com>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Bringing up Elastic also reminds me that the
> > >>> Elastic
> > >>>>>>>> framework
> > >>>>>>>>>> has
> > >>>>>>>>>>>> just
> > >>>>>>>>>>>>>>> recently transitioned out of Open Source, so to
> > >>>>> acknowledge
> > >>>>>>>>> that,
> > >>>>>>>>>>>> maybe
> > >>>>>>>>>>>>>>> some effort toward OpenSearch--I say this not
> > >>>>> understanding
> > >>>>>>>>>> exactly
> > >>>>>>>>>>>> how
> > >>>>>>>>>>>>>>> this sort of thing is considered in a
> > >> large-scale,
> > >>>>>>>> world-class
> > >>>>>>>>>>>> software
> > >>>>>>>>>>>>>>> project like Apache NiFi. (I'm not a contributor,
> > >>>> just
> > >>>>> a
> > >>>>>>>>> grateful
> > >>>>>>>>>>>>>>> consumer.)
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Russ
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On 7/23/21 10:28 AM, Matt Burgess wrote:
> > >>>>>>>>>>>>>>> Along with the itemized list for ancient
> > >> components
> > >>>> we
> > >>>>>>> should
> > >>>>>>>>>> look at
> > >>>>>>>>>>>>>>> updating versions of drivers, SDKs, etc. for
> > >>> external
> > >>>>>>> systems
> > >>>>>>>>>> such as
> > >>>>>>>>>>>>>>> Elasticsearch, Cassandra, etc. There may be
> > >>> breaking
> > >>>>>>> changes
> > >>>>>>>>> but
> > >>>>>>>>>> 2.0
> > >>>>>>>>>>>>>>> is probably the right time to get things up to
> > >> date
> > >>>> to
> > >>>>> make
> > >>>>>>>>> them
> > >>>>>>>>>> more
> > >>>>>>>>>>>>>>> useful to more people.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > >>>>>>>>>> thenatog@gmail.com
> > >>>>>>>>>>>>>> <mailto:
> > >>>>>>>>>>>>>>> thenatog@gmail.com>> wrote:
> > >>>>>>>>>>>>>>> I'm a +1 for removing pretty much all of this
> > >>> stuff.
> > >>>>> There
> > >>>>>>>> are
> > >>>>>>>>>>>> security
> > >>>>>>>>>>>>>>> implications to keeping old dependencies around,
> > >> so
> > >>>> the
> > >>>>>>> more
> > >>>>>>>>> old
> > >>>>>>>>>> code
> > >>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>> can remove the better. I agree that eventually we
> > >>>> need
> > >>>>> to
> > >>>>>>>> move
> > >>>>>>>>> to
> > >>>>>>>>>>>>>>> supporting only Java 11+, and as our next release
> > >>>> will
> > >>>>>>>> probably
> > >>>>>>>>>> be
> > >>>>>>>>>>>>> about
> > >>>>>>>>>>>>>> 4
> > >>>>>>>>>>>>>>> - 6 months from now that doesn't seem too soon.
> > >> We
> > >>>>> could
> > >>>>>>>>>> potentially
> > >>>>>>>>>>>>>> break
> > >>>>>>>>>>>>>>> this in two and remove the deprecated processors
> > >>> and
> > >>>>> leave
> > >>>>>>>> 1.x
> > >>>>>>>>> on
> > >>>>>>>>>>>> Java
> > >>>>>>>>>>>>> 8,
> > >>>>>>>>>>>>>>> and finally start on 2.x which would support only
> > >>>> Java
> > >>>>> 11.
> > >>>>>>>> I'm
> > >>>>>>>>>> unsure
> > >>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>> what implications changing the date and time
> > >>> handling
> > >>>>> would
> > >>>>>>>>> have
> > >>>>>>>>>> -
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>>>>> running systems that use long term historical
> > >> logs,
> > >>>>>>>> unexpected
> > >>>>>>>>>>>> impacts
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> time logging could be a problem.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> As Joe says I think feature work will have to be
> > >>>>> dedicated
> > >>>>>>> to
> > >>>>>>>>>> 2.x and
> > >>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>> could support 1.x for security fixes for some
> > >>> period
> > >>>> of
> > >>>>>>> time.
> > >>>>>>>>> 2.x
> > >>>>>>>>>>>> seems
> > >>>>>>>>>>>>>>> like a gargantuan task but it's probably time to
> > >>> get
> > >>>>>>> started.
> > >>>>>>>>> Not
> > >>>>>>>>>>>> sure
> > >>>>>>>>>>>>>> how
> > >>>>>>>>>>>>>>> we handle all open PRs and the transition between
> > >>> 1.x
> > >>>>> and
> > >>>>>>>> 2.x.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> > >>>>>>>> joe.witt@gmail.com
> > >>>>>>>>>>>> <mailto:
> > >>>>>>>>>>>>>>> joe.witt@gmail.com>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Jon
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> You're right we have to be careful and you're
> > >> right
> > >>>>> there
> > >>>>>>> are
> > >>>>>>>>>> still
> > >>>>>>>>>>>>>>> significant Java 8 users out there. But we also
> > >>> have
> > >>>> to
> > >>>>> be
> > >>>>>>>>>> careful
> > >>>>>>>>>>>>>>> about security and sustainability of the
> > >> codebase.
> > >>> If
> > >>>>> we
> > >>>>>>> had
> > >>>>>>>>>> talked
> > >>>>>>>>>>>>>>> about this last year when that article came out
> > >> I'd
> > >>>>> have
> > >>>>>>>> agreed
> > >>>>>>>>>> it is
> > >>>>>>>>>>>>>>> too early. Interestingly that link seems to get
> > >>>> updated
> > >>>>>>> and I
> > >>>>>>>>>> tried
> > >>>>>>>>>>>>>>> [1] and found more recent data (not sure how
> > >>> recent).
> > >>>>>>> Anyway
> > >>>>>>>> it
> > >>>>>>>>>>>>>>> suggests Java 8 is still the top dog but we see
> > >>> good
> > >>>>> growth
> > >>>>>>>> on
> > >>>>>>>>>> 11. In
> > >>>>>>>>>>>>>>> my $dayjob this aligns to what I'm seeing too.
> > >>>>> Customers
> > >>>>>>>> didn't
> > >>>>>>>>>> seem
> > >>>>>>>>>>>>>>> to care about Java 11 until later half last year
> > >>> and
> > >>>>> now
> > >>>>>>>>>> suddenly it
> > >>>>>>>>>>>>>>> is all over the place.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I think once we put out a NiFi 2.0 release we'd
> > >> see
> > >>>>> rapid
> > >>>>>>>>>> decrease in
> > >>>>>>>>>>>>>>> work on the 1.x line just being blunt. We did
> > >> this
> > >>>> many
> > >>>>>>> years
> > >>>>>>>>> ago
> > >>>>>>>>>>>>>>> with 0.x to 1.x and we stood behind 0.x for a
> > >> while
> > >>>>> (maybe
> > >>>>>>> a
> > >>>>>>>>>> year or
> > >>>>>>>>>>>>>>> so) but it was purely bug fix/security related
> > >>> bits.
> > >>>> We
> > >>>>>>> would
> > >>>>>>>>>> need to
> > >>>>>>>>>>>>>>> do something similar. But feature work would
> > >> almost
> > >>>>>>> certainly
> > >>>>>>>>> go
> > >>>>>>>>>> to
> > >>>>>>>>>>>>>>> the 2.x line. Maybe there are other workable
> > >> models
> > >>>> but
> > >>>>> my
> > >>>>>>>>>> instinct
> > >>>>>>>>>>>>>>> suggests this is likely to follow a similar path.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> ...anyway I agree it isn't that easy of a call to
> > >>>> dump
> > >>>>> Java
> > >>>>>>>> 8.
> > >>>>>>>>> We
> > >>>>>>>>>>>>>>> need to make the call in both the interests of
> > >> the
> > >>>> user
> > >>>>>>> base
> > >>>>>>>>> and
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>> contributor base of the community.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> [1]
> > >>>> https://www.jetbrains.com/lp/devecosystem-2021/java/
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks
> > >>>>>>>>>>>>>>> Joe
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
> > >>>>>>> joe.witt@gmail.com
> > >>>>>>>>>> <mailto:
> > >>>>>>>>>>>>>>> joe.witt@gmail.com>> wrote:
> > >>>>>>>>>>>>>>> Russ
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Yeah the flow registry is a key part of it. But
> > >>> also
> > >>>>> now
> > >>>>>>> you
> > >>>>>>>>> can
> > >>>>>>>>>>>>>>> download the flow definition in JSON (upload i
> > >>> think
> > >>>> is
> > >>>>>>> there
> > >>>>>>>>> now
> > >>>>>>>>>>>>>>> too). Templates offered a series of challenges
> > >> such
> > >>>> as
> > >>>>> we
> > >>>>>>>> store
> > >>>>>>>>>> them
> > >>>>>>>>>>>>>>> in the flow definition which has made flows
> > >> massive
> > >>>> in
> > >>>>> an
> > >>>>>>>>>> unintended
> > >>>>>>>>>>>>>>> way which isn't fun for cluster behavior.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> We have a couple cases where we headed down a
> > >>>>> particular
> > >>>>>>>>> concept
> > >>>>>>>>>> and
> > >>>>>>>>>>>>>>> came up with better approaches later. We need to
> > >>>>> reconcile
> > >>>>>>>>> these
> > >>>>>>>>>> with
> > >>>>>>>>>>>>>>> the benefit of hindsight, and while being careful
> > >>> to
> > >>>> be
> > >>>>> not
> > >>>>>>>>>> overly
> > >>>>>>>>>>>>>>> disruptive to existing users, to reduce the
> > >>>>>>>>> codebase/maintenance
> > >>>>>>>>>>>>>>> burden and allow continued evolution of the
> > >>> project.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > >>>>>>>>>>>> russ@windofkeltia.com
> > >>>>>>>>>>>>>>> <ma...@windofkeltia.com>>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>> Joe,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I apologize for the off-topic intrusion, but what
> > >>>>> replaces
> > >>>>>>>>>> templates?
> > >>>>>>>>>>>>>>> The Registry? Templates rocked and we have used
> > >>> them
> > >>>>> since
> > >>>>>>>>> 0.5.x.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Russ
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On 7/23/21 8:31 AM, Joe Witt wrote:
> > >>>>>>>>>>>>>>> David,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I think this is a highly reasonable approach and
> > >>>> such a
> > >>>>>>> focus
> > >>>>>>>>>> will
> > >>>>>>>>>>>>>>> greatly help make a 2.0 release far more
> > >>> approachable
> > >>>>> to
> > >>>>>>>> knock
> > >>>>>>>>>> out.
> > >>>>>>>>>>>>>>> Not only that but tech debt reduction would help
> > >>> make
> > >>>>> work
> > >>>>>>>>>> towards
> > >>>>>>>>>>>>>>> major features we'd think about in a 'major
> > >>> release'
> > >>>>> sense
> > >>>>>>>> more
> > >>>>>>>>>>>>>>> approachable.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> We should remove all deprecated things (as well
> > >> as
> > >>>>> verify
> > >>>>>>> we
> > >>>>>>>>>> have the
> > >>>>>>>>>>>>>>> right list). We should remove/consider removal of
> > >>>>>>> deprecated
> > >>>>>>>>>>>>>>> concepts
> > >>>>>>>>>>>>>>> like templates. We should consider whether we can
> > >>>>> resolve
> > >>>>>>> the
> > >>>>>>>>>>>>>>> various
> > >>>>>>>>>>>>>>> ways we've handled what are now parameters down
> > >> to
> > >>>> one
> > >>>>>>> clean
> > >>>>>>>>>>>>>>> approach.
> > >>>>>>>>>>>>>>> We should remove options in the nifi.properties
> > >>> which
> > >>>>> turn
> > >>>>>>>> out
> > >>>>>>>>> to
> > >>>>>>>>>>>>>>> never be used quite right (if there are). There
> > >> is
> > >>>>> quite a
> > >>>>>>>> bit
> > >>>>>>>>> we
> > >>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>> do purely in the name of tech debt reduction.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Lots to consider here but I think this is the
> > >> right
> > >>>>>>>> discussion.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Than ks
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> > >>>>>>>> bbende@gmail.com
> > >>>>>>>>>>>> <mailto:
> > >>>>>>>>>>>>>>> bbende@gmail.com>>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>> I'm a +1 for this... Not sure if this falls under
> > >>>>> "Removing
> > >>>>>>>>>>>>>>> Deprecated
> > >>>>>>>>>>>>>>> Components", but I think we should also look at
> > >>>>> anything
> > >>>>>>> that
> > >>>>>>>>> has
> > >>>>>>>>>>>>>>> been
> > >>>>>>>>>>>>>>> marked as deprecated throughout the code base as
> > >> a
> > >>>>>>> candidate
> > >>>>>>>>> for
> > >>>>>>>>>>>>>>> removal. There are quite a few classes, methods,
> > >>>>>>> properties,
> > >>>>>>>>> etc
> > >>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>> have been waiting for a chance to be removed.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > >>>>>>>>>>>>>>> <exceptionfactory@apache.org<mailto:
> > >>>>>>>>> exceptionfactory@apache.org
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>> Team,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> With all of the excellent work that many have
> > >>>>> contributed
> > >>>>>>> to
> > >>>>>>>>> NiFi
> > >>>>>>>>>>>>>>> over the
> > >>>>>>>>>>>>>>> years, the code base has also accumulated some
> > >>> amount
> > >>>>> of
> > >>>>>>>>>> technical
> > >>>>>>>>>>>>>>> debt. A
> > >>>>>>>>>>>>>>> handful of components have been marked as
> > >>> deprecated,
> > >>>>> and
> > >>>>>>>> some
> > >>>>>>>>>>>>>>> components
> > >>>>>>>>>>>>>>> remain in the code base to support integration
> > >> with
> > >>>> old
> > >>>>>>>>> versions
> > >>>>>>>>>>>>>>> of various
> > >>>>>>>>>>>>>>> products. Following the principles of semantic
> > >>>>> versioning,
> > >>>>>>>>>>>>>>> introducing a
> > >>>>>>>>>>>>>>> major release would provide the opportunity to
> > >>> remove
> > >>>>> these
> > >>>>>>>>>>>>>>> deprecated and
> > >>>>>>>>>>>>>>> unsupported components.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Rather than focusing the next major release on
> > >> new
> > >>>>>>> features,
> > >>>>>>>>> what
> > >>>>>>>>>>>>>>> do you
> > >>>>>>>>>>>>>>> think about focusing on technical debt removal?
> > >>> This
> > >>>>>>> approach
> > >>>>>>>>>>>>>>> would not
> > >>>>>>>>>>>>>>> make for the most interesting release, but it
> > >>>> provides
> > >>>>> the
> > >>>>>>>>>>>>>>> opportunity to
> > >>>>>>>>>>>>>>> clean up elements that involve breaking changes.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Focusing on technical debt, at least three
> > >> primary
> > >>>>> goals
> > >>>>>>> come
> > >>>>>>>>> to
> > >>>>>>>>>>>>>>> mind for
> > >>>>>>>>>>>>>>> the next major release:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 1. Removal of deprecated and unmaintained
> > >>> components
> > >>>>>>>>>>>>>>> 2. Require Java 11 as the minimum supported
> > >> version
> > >>>>>>>>>>>>>>> 3. Transition internal date and time handling to
> > >>> JSR
> > >>>>> 310
> > >>>>>>>>>> java.time
> > >>>>>>>>>>>>>>> components
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> *Removing Deprecated Components*
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Removing support for older and deprecated
> > >>> components
> > >>>>>>>> provides a
> > >>>>>>>>>>>>>>> great
> > >>>>>>>>>>>>>>> opportunity to improve the overall security
> > >> posture
> > >>>>> when it
> > >>>>>>>>> comes
> > >>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> maintaining dependencies. The OWASP dependency
> > >>> plugin
> > >>>>>>> report
> > >>>>>>>>>>>>>>> currently
> > >>>>>>>>>>>>>>> generates 50 MB of HTML for questionable
> > >>>> dependencies,
> > >>>>> many
> > >>>>>>>> of
> > >>>>>>>>>>>>>>> which are
> > >>>>>>>>>>>>>>> related to old versions of various libraries.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> As a starting point, here are a handful of
> > >>> components
> > >>>>> and
> > >>>>>>>>>>>>>>> extension modules
> > >>>>>>>>>>>>>>> that could be targeted for removal in a major
> > >>>> version:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> - PostHTTP and GetHTTP
> > >>>>>>>>>>>>>>> - ListenLumberjack and the entire
> > >>>>> nifi-lumberjack-bundle
> > >>>>>>>>>>>>>>> - ListenBeats and the entire nifi-beats-bundle
> > >>>>>>>>>>>>>>> - Elasticsearch 5 components
> > >>>>>>>>>>>>>>> - Hive 1 and 2 components
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> *Requiring Java 11*
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Java 8 is now over seven years old, and NiFi has
> > >>>>> supported
> > >>>>>>>>>> general
> > >>>>>>>>>>>>>>> compatibility with Java 11 for several years.
> > >> NiFi
> > >>>>> 1.14.0
> > >>>>>>>>>>>>>>> incorporated
> > >>>>>>>>>>>>>>> internal improvements specifically related to TLS
> > >>>> 1.3,
> > >>>>>>> which
> > >>>>>>>>>>>>>>> allowed
> > >>>>>>>>>>>>>>> closing out the long-running Java 11
> > >> compatibility
> > >>>> epic
> > >>>>>>>>>> NIFI-5174.
> > >>>>>>>>>>>>>>> Making
> > >>>>>>>>>>>>>>> Java 11 the minimum required version provides the
> > >>>>>>> opportunity
> > >>>>>>>>> to
> > >>>>>>>>>>>>>>> address
> > >>>>>>>>>>>>>>> any lingering edge cases and put NiFi in a better
> > >>>>> position
> > >>>>>>> to
> > >>>>>>>>>>>>>>> support
> > >>>>>>>>>>>>>>> current Java versions.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> *JSR 310 for Date and Time Handling*
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Without making the scope too broad, transitioning
> > >>>>> internal
> > >>>>>>>> date
> > >>>>>>>>>>>>>>> and time
> > >>>>>>>>>>>>>>> handling to use DateTimeFormatter instead of
> > >>>>>>> SimpleDateFormat
> > >>>>>>>>>>>>>>> would provide
> > >>>>>>>>>>>>>>> a number of advantages. The Java Time components
> > >>>>> provide
> > >>>>>>> much
> > >>>>>>>>>>>>>>> better
> > >>>>>>>>>>>>>>> clarity when it comes to handling localized date
> > >>> and
> > >>>>> time
> > >>>>>>>>>>>>>>> representations,
> > >>>>>>>>>>>>>>> and also avoid the inherent confusion of
> > >>>> java.sql.Date
> > >>>>>>>>> extending
> > >>>>>>>>>>>>>>> java.util.Date. Many internal components,
> > >>>> specifically
> > >>>>>>>>>>>>>>> Record-oriented
> > >>>>>>>>>>>>>>> processors and services, rely on date parsing,
> > >>>> leading
> > >>>>> to
> > >>>>>>>>>>>>>>> confusion and
> > >>>>>>>>>>>>>>> various workarounds. The pattern formats of
> > >>>>>>> SimpleDateFormat
> > >>>>>>>>> and
> > >>>>>>>>>>>>>>> DateTimeFormatter are very similar, but there
> > >> are a
> > >>>> few
> > >>>>>>>> subtle
> > >>>>>>>>>>>>>>> differences.
> > >>>>>>>>>>>>>>> Making this transition would provide a much
> > >> better
> > >>>>>>> foundation
> > >>>>>>>>>>>>>>> going forward.
> > >>>>>>>>>>>>>>> *Conclusion*
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks for giving this proposal some
> > >> consideration.
> > >>>>> Many of
> > >>>>>>>> you
> > >>>>>>>>>>>>>>> have been
> > >>>>>>>>>>>>>>> developing NiFi for years and I look forward to
> > >>> your
> > >>>>>>>> feedback.
> > >>>>>>>>> I
> > >>>>>>>>>>>>>>> would be
> > >>>>>>>>>>>>>>> glad to put together a more formalized
> > >>> recommendation
> > >>>>> on
> > >>>>>>>>>>>>>>> Confluence and
> > >>>>>>>>>>>>>>> write up Jira epics if this general approach
> > >> sounds
> > >>>>>>> agreeable
> > >>>>>>>>> to
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> community.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>>>> David Handermann
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>
> > >>>
> > >>
> >
>
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Joe Witt <jo...@gmail.com>.
Kevin,
This may well be a very interesting approach which opens up a lot of
options for us in how we tackle a 2.x...
Thanks
On Tue, Aug 3, 2021 at 8:45 AM Kevin Doran <kd...@gmail.com> wrote:
>
> Outside of core NiFi, for MiNiFi, I think it would be beneficial to align MiNiFi Java and MiNiFI C++ to use the same method(s) for command and control / flow deployment. This was discussed when MiNiFi Java was merged into the NiFi codebase [1].
>
> My proposal would be to create a Java implementation of the C2 Protocol used by MiNiFi C++, both server-side and client-side, to allow deploying to MiNiFi Java instances from a centralize flow definition. Thanks to the unification of MiNiFi Java and NiFi, this could likely also be added as a capability of NiFi at that same time.
>
> If we are able to do this on the 1.x line, then I would also support deprecating other methods of flow deployment to MiNiFi outlined in [1] and removing them in a 2.x release.
>
> [1] https://github.com/apache/nifi/pull/4933#issuecomment-808411977 <https://github.com/apache/nifi/pull/4933#issuecomment-808411977>
> [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design <https://cwiki.apache.org/confluence/display/MINIFI/C2+Design>
>
> Thanks,
> Kevin
>
> > On Aug 3, 2021, at 10:07 AM, David Handermann <ex...@gmail.com> wrote:
> >
> > Thanks for the continued discussion around what can or should be removed.
> > Having a clear migration path from version 1 to version 2 will be very
> > important for supporting current deployments.
> >
> > Following the example of some other projects, one way to consider the
> > upgrade is from the point of view of deprecation warnings. If implemented
> > correctly, an installation of NiFi running the latest minor release of
> > version 1, and showing no deprecation warnings, should be upgradeable to
> > version 2 without any changes. Making this work involves accurately tagging
> > components and methods with deprecation indicators, and providing a clear
> > way to observe deprecation warnings. With the current version containing
> > both deprecated components and deprecated methods in various classes, this
> > would involve some work to get right. A simple approach might be a named
> > logger for deprecation warnings, which would write a separate log file. A
> > more advanced approach might involve a tool to analyze the system and flow
> > configurations. Either way, it seems that some additional work in a minor
> > release version is necessary before considering a major release version.
> >
> > One other point on compatibility: as long as the core component API
> > definition remains backwards compatible, it should be possible to run older
> > components. This provides a transition option for customers using
> > components such as PostHTTP, or any others that are not actively
> > maintained. At some point it may become necessary to break compatibility at
> > that level, but at least for version 2, maintaining component API
> > compatibility is key.
> >
> > Regards,
> > David Handermann
> >
> > On Sun, Aug 1, 2021 at 10:23 AM Mark Bean <ma...@gmail.com> wrote:
> >
> >> I created a JIRA ticket to investigate and improve the performance of
> >> InvokeHTTP. It includes a flow definition for benchmarking the performance
> >> of both PostHTTP and InvokeHTTP.
> >>
> >> https://issues.apache.org/jira/browse/NIFI-8968
> >>
> >> Thanks,
> >> Mark
> >>
> >> On Fri, Jul 30, 2021 at 6:41 PM Adam Taft <ad...@adamtaft.com> wrote:
> >>
> >>> I'm not seeing the side thread that was going to discuss deprecation of
> >>> PostHTTP. Has that thread started and I just don't see it?
> >>>
> >>> One (significant?) concern with PostHTTP is the smooth integration of
> >>> NiFi-to-NiFi communication that is very transparently enabled with the
> >>> ListenHTTP and PostHTTP processors. There's some special logic in there
> >> for
> >>> handling flowfiles that InvokeHTTP doesn't really (nor should really)
> >> have.
> >>>
> >>> I know of several (large) NiFi installations that rely on the PostHTTP /
> >>> ListenHTTP combination. It has enabled NiFi to NiFi communication for
> >> folks
> >>> reluctant or unable to enable site-to-site type configuration.
> >>>
> >>> Honestly, I don't know why we'd want to "deprecate" any processor, as
> >>> opposed to just moving it to a new location. If these processors can be
> >>> ported and maintained to whatever the 2.0 API looks like, there's
> >> possibly
> >>> little harm keeping them around.
> >>>
> >>> And by the way, what happened to the "marketplace" concept? Is this being
> >>> considered for 2.0 as well? Because relocating the deprecated processors
> >>> to an external nar might be the best solution. Losing PostHTTP entirely I
> >>> think would be a mistake, but I'd conceptually support relocating it.
> >>>
> >>> Thanks,
> >>>
> >>> /Adam
> >>>
> >>> On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <jo...@gmail.com> wrote:
> >>>
> >>>> Looks like we just need to knock out 5 JIRAs :) [1]
> >>>>
> >>>> I felt like we had a label folks were using at one point but quickly
> >>>> looking revealed nothing exciting. After this confluence page
> >>>> stabilizes a bit we can probably knock out some JIRAs and such.
> >>>>
> >>>> [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
> >>>>
> >>>> On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <ot...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> I find myself wishing I had a list of all the jiras / issues that
> >> have
> >>>>> been put off for a 2.0 release because they required some change or
> >>>> another
> >>>>> :(
> >>>>>
> >>>>> From: Joe Witt <jo...@gmail.com> <jo...@gmail.com>
> >>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> >> dev@nifi.apache.org>
> >>>>> Date: July 27, 2021 at 12:30:35
> >>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> >>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> >>>>>
> >>>>> A few thoughts:
> >>>>>
> >>>>> 1. I would love to see deprecation notices show up in the UI in
> >>>>> various ways to help motivate users to move off things to more
> >>>>> supportable things. That is not a prerequisite for anything happening
> >>>>> however. Just a good feature/nice thing to do for users when someone
> >>>>> is able to tackle it.
> >>>>>
> >>>>> 2. The decision to deprecate something and to further remove it need
> >>>>> not mean there is a superior solution available. If that thing itself
> >>>>> isn't getting the love/attention it needs to be
> >>>>> maintained/supported/healthy going forward that alone is enough to
> >>>>> remove it. That might well be the case with PostHTTP [1] and for
> >>>>> comparison you can see how much effort has gone into InvokeHTTP [2].
> >>>>>
> >>>>> 3. When discussing a 2.0 release each thing we add as a 'must do' the
> >>>>> further away from reality such a release will become. We'll have to
> >>>>> get very specific about 'musts' vs 'wants'.
> >>>>>
> >>>>> [1]
> >>>>>
> >>>>
> >>>
> >> https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
> >>>>> [2]
> >>>>>
> >>>>
> >>>
> >> https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
> >>>>>
> >>>>> On Tue, Jul 27, 2021 at 8:47 AM David Handermann
> >>>>> <ex...@apache.org> wrote:
> >>>>>>
> >>>>>> Thanks Mark, providing a template or comparison statistics with
> >> Java
> >>>>>> versions and component configuration details would be very helpful.
> >>> If
> >>>> it
> >>>>>> is possible to run tests using a public API or deployable service,
> >>> that
> >>>>>> would also help confirm potential differences.
> >>>>>>
> >>>>>> Showing a deprecation notice in the UI could be helpful, perhaps
> >> as a
> >>>>>> configurable option. NIFI-8650 describes a general Flow Analysis
> >>>>>> capability, and it seems like that might be one possible way to
> >>> surface
> >>>>>> deprecation warnings. For something more specific to component
> >>>>> deprecation,
> >>>>>> it seems important to find a balance between making it obvious and
> >>>> making
> >>>>>> it something that ends up getting ignored.
> >>>>>>
> >>>>>> Regards,
> >>>>>> David Handermann
> >>>>>>
> >>>>>> On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <ma...@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>>> I'll start a new thread for PostHTTP when I get a template and/or
> >>>>> detailed
> >>>>>>> stats.
> >>>>>>>
> >>>>>>> I know the deprecation is noted in the documentation. That's a
> >>>>> necessary
> >>>>>>> and minimum level of notification. I was suggesting it be more
> >>>> obvious
> >>>>> in
> >>>>>>> the UI. I think it would be beneficial to somehow be aware of the
> >>>>>>> deprecation status simply by looking at the flow (perhaps even on
> >>> the
> >>>>>>> summary pages too), and without having to open the documentation
> >>> for
> >>>>> every
> >>>>>>> processor to confirm whether or not a component is marked as
> >>>>> deprecated.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Mark
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
> >>>>>>> exceptionfactory@apache.org> wrote:
> >>>>>>>
> >>>>>>>> Mark,
> >>>>>>>>
> >>>>>>>> Thanks for the feedback. It may be better to start a separate
> >>>> thread
> >>>>> on
> >>>>>>>> PostHTTP, but can you provide an example flow demonstrating the
> >>>>>>> performance
> >>>>>>>> differences between PostHTTP and InvokeHTTP?
> >>>>>>>>
> >>>>>>>> PostHTTP uses the Apache HttpComponents library, whereas
> >>> InvokeHTTP
> >>>>> uses
> >>>>>>>> OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates
> >> for
> >>>>> OkHttp,
> >>>>>>>> so it would be important to test with the most recent version.
> >> It
> >>>> is
> >>>>> also
> >>>>>>>> worth noting that test classes for PostHTTP are now integration
> >>>> tests
> >>>>> as
> >>>>>>>> opposed to unit tests, which means they are not executed as
> >> part
> >>> of
> >>>>> the
> >>>>>>>> automated builds.
> >>>>>>>>
> >>>>>>>> The NiFi documentation includes the contents of the
> >>>> DeprecationNotice
> >>>>> for
> >>>>>>>> PostHTTP:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> David Handermann
> >>>>>>>>
> >>>>>>>> On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <
> >> mark.o.bean@gmail.com
> >>>>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I'm strongly in favor of reducing tech debt, and moving
> >>>>> deliberately
> >>>>>>> to a
> >>>>>>>>> 2.0 release. I have a concern with just one processor that is
> >>>>> currently
> >>>>>>>>> marked as deprecated: PostHTTP. (I have not evaluated
> >>>> specifically
> >>>>> any
> >>>>>>>>> other deprecated components; I cannot say if there are or are
> >>> not
> >>>>>>> similar
> >>>>>>>>> issues elsewhere.) I understand the rationale for deprecating
> >>>> this
> >>>>>>>>> processor in that it eliminates a processor whose
> >> functionality
> >>>> is
> >>>>>>>>> available elsewhere, specifically in the more flexible
> >>>> InvokeHTTP.
> >>>>>>>> However,
> >>>>>>>>> in my experience and testing, PostHTTP performs transfers
> >> with
> >>>> far
> >>>>>>>> greater
> >>>>>>>>> throughput than InvokeHTTP. I would not be in favor of
> >> removing
> >>>>>>> PostHTTP
> >>>>>>>>> unless/until InvokeHTTP is refactored to increase its
> >>> throughput
> >>>>>>>>> capability.
> >>>>>>>>>
> >>>>>>>>> Has anyone else continued to use PostHTTP over InvokeHTTP for
> >>>>> similar
> >>>>>>>>> reasons? Or, is there a performance-related configuration for
> >>>>>>> InvokeHTTP
> >>>>>>>> I
> >>>>>>>>> may have missed?
> >>>>>>>>>
> >>>>>>>>> Also, in order to help facilitate a smooth transition to 2.0
> >>>> from a
> >>>>>>> user
> >>>>>>>>> perspective, would it be advisable to add some sort of visual
> >>>>> indicator
> >>>>>>>> in
> >>>>>>>>> the UI for components that are currently annotated as
> >>>> @Deprecated?
> >>>>> This
> >>>>>>>>> might help users proactively modify their flow prior to a
> >>> release
> >>>>> that
> >>>>>>>>> would otherwise break it.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <
> >> bbende@gmail.com>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> With the merging of MiNiFi and registry into the NiFi repo,
> >>>> we've
> >>>>>>>>>> actually gone more towards a mono-repo where everything is
> >>>>> released
> >>>>>>>>>> together. Eventually it would still be nice to have a
> >> smaller
> >>>>> base
> >>>>>>>>>> distribution containing just the framework and standard
> >> NARs,
> >>>> but
> >>>>> it
> >>>>>>>>>> is hard to tackle that until we provide a way for users to
> >>>> easily
> >>>>>>>>>> obtain the NARs in some other way.
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <
> >>>>> edward.armes@gmail.com
> >>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Given the major version number shift and the spliting up
> >> of
> >>>>>>>> processors
> >>>>>>>>>> into
> >>>>>>>>>>> multiple NAR's. I'd like to suggest that we start
> >>>> individually
> >>>>>>>>> versioning
> >>>>>>>>>>> individual NARs/ bundles.
> >>>>>>>>>>>
> >>>>>>>>>>> I can see this bringing a large number of benifits
> >>> including
> >>>>> making
> >>>>>>>>> Nifi
> >>>>>>>>>>> more deployable with things RPM, but also potentially
> >>>> allowing
> >>>>>>> those
> >>>>>>>>> that
> >>>>>>>>>>> have to deploy managed Nifi instances easier to upgrade.
> >>>>>>>>>>>
> >>>>>>>>>>> Edward
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, 26 Jul 2021, 20:42 Otto Fowler, <
> >>>> ottobackwards@gmail.com>
> >>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> The issue with updating the aws sdk is if it breaks any
> >>> one
> >>>>> of
> >>>>>>> the
> >>>>>>>>>>>> processors.
> >>>>>>>>>>>> the Web Gateway API invoke processor for example is not
> >>>> using
> >>>>> a
> >>>>>>>> high
> >>>>>>>>>> level
> >>>>>>>>>>>> purpose build client and may break.
> >>>>>>>>>>>>
> >>>>>>>>>>>> If we change the aws version, we need to coordinate in
> >>>> such a
> >>>>> way
> >>>>>>>>> that
> >>>>>>>>>> they
> >>>>>>>>>>>> all
> >>>>>>>>>>>> can come along reasonably.
> >>>>>>>>>>>> IE: what happens if 1 or 2 break but the rest or OK?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> From: David Handermann <ex...@apache.org>
> >>>>>>>>>>>> <ex...@apache.org>
> >>>>>>>>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> >>>>>>>>> dev@nifi.apache.org>
> >>>>>>>>>>>> Date: July 26, 2021 at 09:33:42
> >>>>>>>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
> >>>>>>> dev@nifi.apache.org
> >>>>>>>>>
> >>>>>>>>>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> >>>>>>>>>>>>
> >>>>>>>>>>>> Chris,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the reply and recommendations. It seems like
> >>>> some
> >>>>> of
> >>>>>>> the
> >>>>>>>>>> work to
> >>>>>>>>>>>> reorganize the module structure could be done outside
> >> of
> >>> a
> >>>>> major
> >>>>>>>>>> release,
> >>>>>>>>>>>> but it would be great to target any breaking changes
> >> for
> >>>> 2.0.
> >>>>>>>>> Perhaps a
> >>>>>>>>>>>> separate feature proposal on module restructuring, with
> >>> the
> >>>>> goal
> >>>>>>> of
> >>>>>>>>>>>> supporting optimized builds, would be a helpful way to
> >>> move
> >>>>> that
> >>>>>>>> part
> >>>>>>>>>> of
> >>>>>>>>>>>> the discussion forward.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Regarding updating AWS SDK to version 2, it seems like
> >>> that
> >>>>> might
> >>>>>>>> be
> >>>>>>>>>>>> possible now. I haven't taken a close look at the
> >>>> referencing
> >>>>>>>>>> components,
> >>>>>>>>>>>> so I'm not sure about the level of effort involved.
> >> Minor
> >>>>> NiFi
> >>>>>>>>> version
> >>>>>>>>>>>> updates have incorporated major new versions of
> >>>> dependencies.
> >>>>> For
> >>>>>>>>>> example,
> >>>>>>>>>>>> NiFi 1.14 included an upgrade from Spring Framework 4
> >> to
> >>> 5.
> >>>>> On
> >>>>>>> the
> >>>>>>>>> one
> >>>>>>>>>>>> hand, including the AWS SDK update as part of a major
> >>>> release
> >>>>>>> seems
> >>>>>>>>>>>> helpful, but unless there are changes that break
> >> existing
> >>>>>>> component
> >>>>>>>>>>>> properties, upgrading the AWS SDK could be worked
> >>>>> independently.
> >>>>>>>>>> Others may
> >>>>>>>>>>>> have more insight into particular usage of that
> >> library.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Regards,
> >>>>>>>>>>>> David Handermann
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> >>>>>>>>>>>> <ch...@naimuri.com.invalid> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Might be worth considering refactoring the build as
> >>> part
> >>>> of
> >>>>>>> this
> >>>>>>>>> work
> >>>>>>>>>>>> too,
> >>>>>>>>>>>>> e.g. only building the bits of the repo affected by a
> >>>>> commit,
> >>>>>>>> etc.
> >>>>>>>>> -
> >>>>>>>>>>>>> discussed briefly in previous threads but don't think
> >>> any
> >>>>>>> changes
> >>>>>>>>>> made
> >>>>>>>>>>>> yet.
> >>>>>>>>>>>>> If NARs/components are likely to be split up and
> >>>> refactored
> >>>>>>> then
> >>>>>>>>> such
> >>>>>>>>>>>> work
> >>>>>>>>>>>>> around the build probably makes sense to consider.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I've a couple of PRs open that include updates to
> >>>>> Elasticsearch
> >>>>>>>>>> versions
> >>>>>>>>>>>>> already, although I stopped at 7.10.2 (after which
> >>>> Elastic
> >>>>>>>> changed
> >>>>>>>>>>>> licence)
> >>>>>>>>>>>>> in case there were licence concerns. But more work
> >> can
> >>> be
> >>>>> done
> >>>>>>> to
> >>>>>>>>>> tidy up
> >>>>>>>>>>>>> the processors, absolutely.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> AWS libraries to v2 would seem a sensible move and a
> >>>>> refactor
> >>>>>>> of
> >>>>>>>>>> those
> >>>>>>>>>>>>> processors as well.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Chris Sampson
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Sat, 24 Jul 2021, 17:47 David Handermann, <
> >>>>>>>>>>>> exceptionfactory@apache.org>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for pointing out the standard NAR bundles,
> >>> Mark.
> >>>>> There
> >>>>>>>>> are a
> >>>>>>>>>>>>> number
> >>>>>>>>>>>>>> of components in the standard NAR bundles with
> >>>> particular
> >>>>>>>>>> dependencies
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>> would make more sense in separate NARs.
> >> Reorganizing
> >>>> the
> >>>>>>>> standard
> >>>>>>>>>> NAR
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> components with limited dependencies and wide
> >>>>> applicability
> >>>>>>>> would
> >>>>>>>>>>>>>> definitely help with future maintenance.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>> David Handermann
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> >>>>>>>>> markap14@hotmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> There’s also some code that exists in order to
> >>>> maintain
> >>>>>>>>> backward
> >>>>>>>>>>>>>>> compatibility in the repositories. I would very
> >>> much
> >>>>> like
> >>>>>>> the
> >>>>>>>>>>>>>> repositories
> >>>>>>>>>>>>>>> to contain no unnecessary code. And swap file
> >>> format
> >>>>>>> supports
> >>>>>>>>>> really
> >>>>>>>>>>>>> old
> >>>>>>>>>>>>>>> formats. And the old impls of the repositories
> >>>>> themselves,
> >>>>>>>> like
> >>>>>>>>>>>>>>> PersistentProvRepo instead of WriteAheadProv
> >> Repo,
> >>>> etc.
> >>>>>>> Lots
> >>>>>>>> of
> >>>>>>>>>> stuff
> >>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>> that can be removed. And some methods in
> >>>> ProcessSession
> >>>>>>> that
> >>>>>>>>> are
> >>>>>>>>>>>> never
> >>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>> by any processor in the codebase but exists in
> >> the
> >>>>> public
> >>>>>>> API
> >>>>>>>>> so
> >>>>>>>>>>>> can’t
> >>>>>>>>>>>>> be
> >>>>>>>>>>>>>>> removed till 2.0.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I think his is also a great time to clean up the
> >>>>> “standard
> >>>>>>>>> nar.”
> >>>>>>>>>> At
> >>>>>>>>>>>>> this
> >>>>>>>>>>>>>>> point, it’s something like 70 MB. And many of the
> >>>>>>> components
> >>>>>>>>>> there
> >>>>>>>>>>>> are
> >>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>> really “standard” - things like connecting to
> >> FTP &
> >>>>> SFTP
> >>>>>>>>>> servers, XML
> >>>>>>>>>>>>>>> processing, Jolt transform, etc. could
> >> potentially
> >>> be
> >>>>> moved
> >>>>>>>>> into
> >>>>>>>>>>>> other
> >>>>>>>>>>>>>>> nars. The
> >>>>> nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
> >>>>>>> is
> >>>>>>>>>> 6.9 MB
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>> necessary for stateless or minifi java. Lots of
> >>>> things
> >>>>>>>> probably
> >>>>>>>>>> to
> >>>>>>>>>>>>>>> reconsider within the standard nar.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I definitely think this is a reasonable approach,
> >>> to
> >>>>> allow
> >>>>>>>> for
> >>>>>>>>> a
> >>>>>>>>>> 2.0
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>> is not a huge feature release but allows the
> >>> project
> >>>> to
> >>>>> be
> >>>>>>>>>> simpler
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>> nimble.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>> -Mark
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> >>>>>>>>>> mikerthomsen@gmail.com
> >>>>>>>>>>>>>> <mailto:
> >>>>>>>>>>>>>>> mikerthomsen@gmail.com>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Russell,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> AFAICT from looking at Elastic's repos, the low
> >>> level
> >>>>> REST
> >>>>>>>>>> client is
> >>>>>>>>>>>>>>> still fine.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Our Elasticsearch support is spread over two NARs
> >>> at
> >>>>>>> present.
> >>>>>>>>> One
> >>>>>>>>>>>> uses
> >>>>>>>>>>>>>>> OkHttp and the other uses that low level Elastic
> >>> REST
> >>>>>>> client.
> >>>>>>>>>>>>>>> Therefore, I think we're fine on licensing for
> >> the
> >>>>> moment.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Mike
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> >>>>>>>>>>>> russ@windofkeltia.com
> >>>>>>>>>>>>>>> <ma...@windofkeltia.com>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Bringing up Elastic also reminds me that the
> >>> Elastic
> >>>>>>>> framework
> >>>>>>>>>> has
> >>>>>>>>>>>> just
> >>>>>>>>>>>>>>> recently transitioned out of Open Source, so to
> >>>>> acknowledge
> >>>>>>>>> that,
> >>>>>>>>>>>> maybe
> >>>>>>>>>>>>>>> some effort toward OpenSearch--I say this not
> >>>>> understanding
> >>>>>>>>>> exactly
> >>>>>>>>>>>> how
> >>>>>>>>>>>>>>> this sort of thing is considered in a
> >> large-scale,
> >>>>>>>> world-class
> >>>>>>>>>>>> software
> >>>>>>>>>>>>>>> project like Apache NiFi. (I'm not a contributor,
> >>>> just
> >>>>> a
> >>>>>>>>> grateful
> >>>>>>>>>>>>>>> consumer.)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Russ
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 7/23/21 10:28 AM, Matt Burgess wrote:
> >>>>>>>>>>>>>>> Along with the itemized list for ancient
> >> components
> >>>> we
> >>>>>>> should
> >>>>>>>>>> look at
> >>>>>>>>>>>>>>> updating versions of drivers, SDKs, etc. for
> >>> external
> >>>>>>> systems
> >>>>>>>>>> such as
> >>>>>>>>>>>>>>> Elasticsearch, Cassandra, etc. There may be
> >>> breaking
> >>>>>>> changes
> >>>>>>>>> but
> >>>>>>>>>> 2.0
> >>>>>>>>>>>>>>> is probably the right time to get things up to
> >> date
> >>>> to
> >>>>> make
> >>>>>>>>> them
> >>>>>>>>>> more
> >>>>>>>>>>>>>>> useful to more people.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> >>>>>>>>>> thenatog@gmail.com
> >>>>>>>>>>>>>> <mailto:
> >>>>>>>>>>>>>>> thenatog@gmail.com>> wrote:
> >>>>>>>>>>>>>>> I'm a +1 for removing pretty much all of this
> >>> stuff.
> >>>>> There
> >>>>>>>> are
> >>>>>>>>>>>> security
> >>>>>>>>>>>>>>> implications to keeping old dependencies around,
> >> so
> >>>> the
> >>>>>>> more
> >>>>>>>>> old
> >>>>>>>>>> code
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>>> can remove the better. I agree that eventually we
> >>>> need
> >>>>> to
> >>>>>>>> move
> >>>>>>>>> to
> >>>>>>>>>>>>>>> supporting only Java 11+, and as our next release
> >>>> will
> >>>>>>>> probably
> >>>>>>>>>> be
> >>>>>>>>>>>>> about
> >>>>>>>>>>>>>> 4
> >>>>>>>>>>>>>>> - 6 months from now that doesn't seem too soon.
> >> We
> >>>>> could
> >>>>>>>>>> potentially
> >>>>>>>>>>>>>> break
> >>>>>>>>>>>>>>> this in two and remove the deprecated processors
> >>> and
> >>>>> leave
> >>>>>>>> 1.x
> >>>>>>>>> on
> >>>>>>>>>>>> Java
> >>>>>>>>>>>>> 8,
> >>>>>>>>>>>>>>> and finally start on 2.x which would support only
> >>>> Java
> >>>>> 11.
> >>>>>>>> I'm
> >>>>>>>>>> unsure
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> what implications changing the date and time
> >>> handling
> >>>>> would
> >>>>>>>>> have
> >>>>>>>>>> -
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>>> running systems that use long term historical
> >> logs,
> >>>>>>>> unexpected
> >>>>>>>>>>>> impacts
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> time logging could be a problem.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> As Joe says I think feature work will have to be
> >>>>> dedicated
> >>>>>>> to
> >>>>>>>>>> 2.x and
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>>> could support 1.x for security fixes for some
> >>> period
> >>>> of
> >>>>>>> time.
> >>>>>>>>> 2.x
> >>>>>>>>>>>> seems
> >>>>>>>>>>>>>>> like a gargantuan task but it's probably time to
> >>> get
> >>>>>>> started.
> >>>>>>>>> Not
> >>>>>>>>>>>> sure
> >>>>>>>>>>>>>> how
> >>>>>>>>>>>>>>> we handle all open PRs and the transition between
> >>> 1.x
> >>>>> and
> >>>>>>>> 2.x.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> >>>>>>>> joe.witt@gmail.com
> >>>>>>>>>>>> <mailto:
> >>>>>>>>>>>>>>> joe.witt@gmail.com>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Jon
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> You're right we have to be careful and you're
> >> right
> >>>>> there
> >>>>>>> are
> >>>>>>>>>> still
> >>>>>>>>>>>>>>> significant Java 8 users out there. But we also
> >>> have
> >>>> to
> >>>>> be
> >>>>>>>>>> careful
> >>>>>>>>>>>>>>> about security and sustainability of the
> >> codebase.
> >>> If
> >>>>> we
> >>>>>>> had
> >>>>>>>>>> talked
> >>>>>>>>>>>>>>> about this last year when that article came out
> >> I'd
> >>>>> have
> >>>>>>>> agreed
> >>>>>>>>>> it is
> >>>>>>>>>>>>>>> too early. Interestingly that link seems to get
> >>>> updated
> >>>>>>> and I
> >>>>>>>>>> tried
> >>>>>>>>>>>>>>> [1] and found more recent data (not sure how
> >>> recent).
> >>>>>>> Anyway
> >>>>>>>> it
> >>>>>>>>>>>>>>> suggests Java 8 is still the top dog but we see
> >>> good
> >>>>> growth
> >>>>>>>> on
> >>>>>>>>>> 11. In
> >>>>>>>>>>>>>>> my $dayjob this aligns to what I'm seeing too.
> >>>>> Customers
> >>>>>>>> didn't
> >>>>>>>>>> seem
> >>>>>>>>>>>>>>> to care about Java 11 until later half last year
> >>> and
> >>>>> now
> >>>>>>>>>> suddenly it
> >>>>>>>>>>>>>>> is all over the place.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I think once we put out a NiFi 2.0 release we'd
> >> see
> >>>>> rapid
> >>>>>>>>>> decrease in
> >>>>>>>>>>>>>>> work on the 1.x line just being blunt. We did
> >> this
> >>>> many
> >>>>>>> years
> >>>>>>>>> ago
> >>>>>>>>>>>>>>> with 0.x to 1.x and we stood behind 0.x for a
> >> while
> >>>>> (maybe
> >>>>>>> a
> >>>>>>>>>> year or
> >>>>>>>>>>>>>>> so) but it was purely bug fix/security related
> >>> bits.
> >>>> We
> >>>>>>> would
> >>>>>>>>>> need to
> >>>>>>>>>>>>>>> do something similar. But feature work would
> >> almost
> >>>>>>> certainly
> >>>>>>>>> go
> >>>>>>>>>> to
> >>>>>>>>>>>>>>> the 2.x line. Maybe there are other workable
> >> models
> >>>> but
> >>>>> my
> >>>>>>>>>> instinct
> >>>>>>>>>>>>>>> suggests this is likely to follow a similar path.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ...anyway I agree it isn't that easy of a call to
> >>>> dump
> >>>>> Java
> >>>>>>>> 8.
> >>>>>>>>> We
> >>>>>>>>>>>>>>> need to make the call in both the interests of
> >> the
> >>>> user
> >>>>>>> base
> >>>>>>>>> and
> >>>>>>>>>> the
> >>>>>>>>>>>>>>> contributor base of the community.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [1]
> >>>> https://www.jetbrains.com/lp/devecosystem-2021/java/
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>> Joe
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
> >>>>>>> joe.witt@gmail.com
> >>>>>>>>>> <mailto:
> >>>>>>>>>>>>>>> joe.witt@gmail.com>> wrote:
> >>>>>>>>>>>>>>> Russ
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Yeah the flow registry is a key part of it. But
> >>> also
> >>>>> now
> >>>>>>> you
> >>>>>>>>> can
> >>>>>>>>>>>>>>> download the flow definition in JSON (upload i
> >>> think
> >>>> is
> >>>>>>> there
> >>>>>>>>> now
> >>>>>>>>>>>>>>> too). Templates offered a series of challenges
> >> such
> >>>> as
> >>>>> we
> >>>>>>>> store
> >>>>>>>>>> them
> >>>>>>>>>>>>>>> in the flow definition which has made flows
> >> massive
> >>>> in
> >>>>> an
> >>>>>>>>>> unintended
> >>>>>>>>>>>>>>> way which isn't fun for cluster behavior.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> We have a couple cases where we headed down a
> >>>>> particular
> >>>>>>>>> concept
> >>>>>>>>>> and
> >>>>>>>>>>>>>>> came up with better approaches later. We need to
> >>>>> reconcile
> >>>>>>>>> these
> >>>>>>>>>> with
> >>>>>>>>>>>>>>> the benefit of hindsight, and while being careful
> >>> to
> >>>> be
> >>>>> not
> >>>>>>>>>> overly
> >>>>>>>>>>>>>>> disruptive to existing users, to reduce the
> >>>>>>>>> codebase/maintenance
> >>>>>>>>>>>>>>> burden and allow continued evolution of the
> >>> project.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> >>>>>>>>>>>> russ@windofkeltia.com
> >>>>>>>>>>>>>>> <ma...@windofkeltia.com>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>> Joe,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I apologize for the off-topic intrusion, but what
> >>>>> replaces
> >>>>>>>>>> templates?
> >>>>>>>>>>>>>>> The Registry? Templates rocked and we have used
> >>> them
> >>>>> since
> >>>>>>>>> 0.5.x.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Russ
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 7/23/21 8:31 AM, Joe Witt wrote:
> >>>>>>>>>>>>>>> David,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I think this is a highly reasonable approach and
> >>>> such a
> >>>>>>> focus
> >>>>>>>>>> will
> >>>>>>>>>>>>>>> greatly help make a 2.0 release far more
> >>> approachable
> >>>>> to
> >>>>>>>> knock
> >>>>>>>>>> out.
> >>>>>>>>>>>>>>> Not only that but tech debt reduction would help
> >>> make
> >>>>> work
> >>>>>>>>>> towards
> >>>>>>>>>>>>>>> major features we'd think about in a 'major
> >>> release'
> >>>>> sense
> >>>>>>>> more
> >>>>>>>>>>>>>>> approachable.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> We should remove all deprecated things (as well
> >> as
> >>>>> verify
> >>>>>>> we
> >>>>>>>>>> have the
> >>>>>>>>>>>>>>> right list). We should remove/consider removal of
> >>>>>>> deprecated
> >>>>>>>>>>>>>>> concepts
> >>>>>>>>>>>>>>> like templates. We should consider whether we can
> >>>>> resolve
> >>>>>>> the
> >>>>>>>>>>>>>>> various
> >>>>>>>>>>>>>>> ways we've handled what are now parameters down
> >> to
> >>>> one
> >>>>>>> clean
> >>>>>>>>>>>>>>> approach.
> >>>>>>>>>>>>>>> We should remove options in the nifi.properties
> >>> which
> >>>>> turn
> >>>>>>>> out
> >>>>>>>>> to
> >>>>>>>>>>>>>>> never be used quite right (if there are). There
> >> is
> >>>>> quite a
> >>>>>>>> bit
> >>>>>>>>> we
> >>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>> do purely in the name of tech debt reduction.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Lots to consider here but I think this is the
> >> right
> >>>>>>>> discussion.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Than ks
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> >>>>>>>> bbende@gmail.com
> >>>>>>>>>>>> <mailto:
> >>>>>>>>>>>>>>> bbende@gmail.com>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>> I'm a +1 for this... Not sure if this falls under
> >>>>> "Removing
> >>>>>>>>>>>>>>> Deprecated
> >>>>>>>>>>>>>>> Components", but I think we should also look at
> >>>>> anything
> >>>>>>> that
> >>>>>>>>> has
> >>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>> marked as deprecated throughout the code base as
> >> a
> >>>>>>> candidate
> >>>>>>>>> for
> >>>>>>>>>>>>>>> removal. There are quite a few classes, methods,
> >>>>>>> properties,
> >>>>>>>>> etc
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>> have been waiting for a chance to be removed.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> >>>>>>>>>>>>>>> <exceptionfactory@apache.org<mailto:
> >>>>>>>>> exceptionfactory@apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>> Team,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> With all of the excellent work that many have
> >>>>> contributed
> >>>>>>> to
> >>>>>>>>> NiFi
> >>>>>>>>>>>>>>> over the
> >>>>>>>>>>>>>>> years, the code base has also accumulated some
> >>> amount
> >>>>> of
> >>>>>>>>>> technical
> >>>>>>>>>>>>>>> debt. A
> >>>>>>>>>>>>>>> handful of components have been marked as
> >>> deprecated,
> >>>>> and
> >>>>>>>> some
> >>>>>>>>>>>>>>> components
> >>>>>>>>>>>>>>> remain in the code base to support integration
> >> with
> >>>> old
> >>>>>>>>> versions
> >>>>>>>>>>>>>>> of various
> >>>>>>>>>>>>>>> products. Following the principles of semantic
> >>>>> versioning,
> >>>>>>>>>>>>>>> introducing a
> >>>>>>>>>>>>>>> major release would provide the opportunity to
> >>> remove
> >>>>> these
> >>>>>>>>>>>>>>> deprecated and
> >>>>>>>>>>>>>>> unsupported components.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Rather than focusing the next major release on
> >> new
> >>>>>>> features,
> >>>>>>>>> what
> >>>>>>>>>>>>>>> do you
> >>>>>>>>>>>>>>> think about focusing on technical debt removal?
> >>> This
> >>>>>>> approach
> >>>>>>>>>>>>>>> would not
> >>>>>>>>>>>>>>> make for the most interesting release, but it
> >>>> provides
> >>>>> the
> >>>>>>>>>>>>>>> opportunity to
> >>>>>>>>>>>>>>> clean up elements that involve breaking changes.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Focusing on technical debt, at least three
> >> primary
> >>>>> goals
> >>>>>>> come
> >>>>>>>>> to
> >>>>>>>>>>>>>>> mind for
> >>>>>>>>>>>>>>> the next major release:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 1. Removal of deprecated and unmaintained
> >>> components
> >>>>>>>>>>>>>>> 2. Require Java 11 as the minimum supported
> >> version
> >>>>>>>>>>>>>>> 3. Transition internal date and time handling to
> >>> JSR
> >>>>> 310
> >>>>>>>>>> java.time
> >>>>>>>>>>>>>>> components
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> *Removing Deprecated Components*
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Removing support for older and deprecated
> >>> components
> >>>>>>>> provides a
> >>>>>>>>>>>>>>> great
> >>>>>>>>>>>>>>> opportunity to improve the overall security
> >> posture
> >>>>> when it
> >>>>>>>>> comes
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> maintaining dependencies. The OWASP dependency
> >>> plugin
> >>>>>>> report
> >>>>>>>>>>>>>>> currently
> >>>>>>>>>>>>>>> generates 50 MB of HTML for questionable
> >>>> dependencies,
> >>>>> many
> >>>>>>>> of
> >>>>>>>>>>>>>>> which are
> >>>>>>>>>>>>>>> related to old versions of various libraries.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> As a starting point, here are a handful of
> >>> components
> >>>>> and
> >>>>>>>>>>>>>>> extension modules
> >>>>>>>>>>>>>>> that could be targeted for removal in a major
> >>>> version:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> - PostHTTP and GetHTTP
> >>>>>>>>>>>>>>> - ListenLumberjack and the entire
> >>>>> nifi-lumberjack-bundle
> >>>>>>>>>>>>>>> - ListenBeats and the entire nifi-beats-bundle
> >>>>>>>>>>>>>>> - Elasticsearch 5 components
> >>>>>>>>>>>>>>> - Hive 1 and 2 components
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> *Requiring Java 11*
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Java 8 is now over seven years old, and NiFi has
> >>>>> supported
> >>>>>>>>>> general
> >>>>>>>>>>>>>>> compatibility with Java 11 for several years.
> >> NiFi
> >>>>> 1.14.0
> >>>>>>>>>>>>>>> incorporated
> >>>>>>>>>>>>>>> internal improvements specifically related to TLS
> >>>> 1.3,
> >>>>>>> which
> >>>>>>>>>>>>>>> allowed
> >>>>>>>>>>>>>>> closing out the long-running Java 11
> >> compatibility
> >>>> epic
> >>>>>>>>>> NIFI-5174.
> >>>>>>>>>>>>>>> Making
> >>>>>>>>>>>>>>> Java 11 the minimum required version provides the
> >>>>>>> opportunity
> >>>>>>>>> to
> >>>>>>>>>>>>>>> address
> >>>>>>>>>>>>>>> any lingering edge cases and put NiFi in a better
> >>>>> position
> >>>>>>> to
> >>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>> current Java versions.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> *JSR 310 for Date and Time Handling*
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Without making the scope too broad, transitioning
> >>>>> internal
> >>>>>>>> date
> >>>>>>>>>>>>>>> and time
> >>>>>>>>>>>>>>> handling to use DateTimeFormatter instead of
> >>>>>>> SimpleDateFormat
> >>>>>>>>>>>>>>> would provide
> >>>>>>>>>>>>>>> a number of advantages. The Java Time components
> >>>>> provide
> >>>>>>> much
> >>>>>>>>>>>>>>> better
> >>>>>>>>>>>>>>> clarity when it comes to handling localized date
> >>> and
> >>>>> time
> >>>>>>>>>>>>>>> representations,
> >>>>>>>>>>>>>>> and also avoid the inherent confusion of
> >>>> java.sql.Date
> >>>>>>>>> extending
> >>>>>>>>>>>>>>> java.util.Date. Many internal components,
> >>>> specifically
> >>>>>>>>>>>>>>> Record-oriented
> >>>>>>>>>>>>>>> processors and services, rely on date parsing,
> >>>> leading
> >>>>> to
> >>>>>>>>>>>>>>> confusion and
> >>>>>>>>>>>>>>> various workarounds. The pattern formats of
> >>>>>>> SimpleDateFormat
> >>>>>>>>> and
> >>>>>>>>>>>>>>> DateTimeFormatter are very similar, but there
> >> are a
> >>>> few
> >>>>>>>> subtle
> >>>>>>>>>>>>>>> differences.
> >>>>>>>>>>>>>>> Making this transition would provide a much
> >> better
> >>>>>>> foundation
> >>>>>>>>>>>>>>> going forward.
> >>>>>>>>>>>>>>> *Conclusion*
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for giving this proposal some
> >> consideration.
> >>>>> Many of
> >>>>>>>> you
> >>>>>>>>>>>>>>> have been
> >>>>>>>>>>>>>>> developing NiFi for years and I look forward to
> >>> your
> >>>>>>>> feedback.
> >>>>>>>>> I
> >>>>>>>>>>>>>>> would be
> >>>>>>>>>>>>>>> glad to put together a more formalized
> >>> recommendation
> >>>>> on
> >>>>>>>>>>>>>>> Confluence and
> >>>>>>>>>>>>>>> write up Jira epics if this general approach
> >> sounds
> >>>>>>> agreeable
> >>>>>>>>> to
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> community.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>> David Handermann
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Kevin Doran <kd...@gmail.com>.
Outside of core NiFi, for MiNiFi, I think it would be beneficial to align MiNiFi Java and MiNiFI C++ to use the same method(s) for command and control / flow deployment. This was discussed when MiNiFi Java was merged into the NiFi codebase [1].
My proposal would be to create a Java implementation of the C2 Protocol used by MiNiFi C++, both server-side and client-side, to allow deploying to MiNiFi Java instances from a centralize flow definition. Thanks to the unification of MiNiFi Java and NiFi, this could likely also be added as a capability of NiFi at that same time.
If we are able to do this on the 1.x line, then I would also support deprecating other methods of flow deployment to MiNiFi outlined in [1] and removing them in a 2.x release.
[1] https://github.com/apache/nifi/pull/4933#issuecomment-808411977 <https://github.com/apache/nifi/pull/4933#issuecomment-808411977>
[2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design <https://cwiki.apache.org/confluence/display/MINIFI/C2+Design>
Thanks,
Kevin
> On Aug 3, 2021, at 10:07 AM, David Handermann <ex...@gmail.com> wrote:
>
> Thanks for the continued discussion around what can or should be removed.
> Having a clear migration path from version 1 to version 2 will be very
> important for supporting current deployments.
>
> Following the example of some other projects, one way to consider the
> upgrade is from the point of view of deprecation warnings. If implemented
> correctly, an installation of NiFi running the latest minor release of
> version 1, and showing no deprecation warnings, should be upgradeable to
> version 2 without any changes. Making this work involves accurately tagging
> components and methods with deprecation indicators, and providing a clear
> way to observe deprecation warnings. With the current version containing
> both deprecated components and deprecated methods in various classes, this
> would involve some work to get right. A simple approach might be a named
> logger for deprecation warnings, which would write a separate log file. A
> more advanced approach might involve a tool to analyze the system and flow
> configurations. Either way, it seems that some additional work in a minor
> release version is necessary before considering a major release version.
>
> One other point on compatibility: as long as the core component API
> definition remains backwards compatible, it should be possible to run older
> components. This provides a transition option for customers using
> components such as PostHTTP, or any others that are not actively
> maintained. At some point it may become necessary to break compatibility at
> that level, but at least for version 2, maintaining component API
> compatibility is key.
>
> Regards,
> David Handermann
>
> On Sun, Aug 1, 2021 at 10:23 AM Mark Bean <ma...@gmail.com> wrote:
>
>> I created a JIRA ticket to investigate and improve the performance of
>> InvokeHTTP. It includes a flow definition for benchmarking the performance
>> of both PostHTTP and InvokeHTTP.
>>
>> https://issues.apache.org/jira/browse/NIFI-8968
>>
>> Thanks,
>> Mark
>>
>> On Fri, Jul 30, 2021 at 6:41 PM Adam Taft <ad...@adamtaft.com> wrote:
>>
>>> I'm not seeing the side thread that was going to discuss deprecation of
>>> PostHTTP. Has that thread started and I just don't see it?
>>>
>>> One (significant?) concern with PostHTTP is the smooth integration of
>>> NiFi-to-NiFi communication that is very transparently enabled with the
>>> ListenHTTP and PostHTTP processors. There's some special logic in there
>> for
>>> handling flowfiles that InvokeHTTP doesn't really (nor should really)
>> have.
>>>
>>> I know of several (large) NiFi installations that rely on the PostHTTP /
>>> ListenHTTP combination. It has enabled NiFi to NiFi communication for
>> folks
>>> reluctant or unable to enable site-to-site type configuration.
>>>
>>> Honestly, I don't know why we'd want to "deprecate" any processor, as
>>> opposed to just moving it to a new location. If these processors can be
>>> ported and maintained to whatever the 2.0 API looks like, there's
>> possibly
>>> little harm keeping them around.
>>>
>>> And by the way, what happened to the "marketplace" concept? Is this being
>>> considered for 2.0 as well? Because relocating the deprecated processors
>>> to an external nar might be the best solution. Losing PostHTTP entirely I
>>> think would be a mistake, but I'd conceptually support relocating it.
>>>
>>> Thanks,
>>>
>>> /Adam
>>>
>>> On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <jo...@gmail.com> wrote:
>>>
>>>> Looks like we just need to knock out 5 JIRAs :) [1]
>>>>
>>>> I felt like we had a label folks were using at one point but quickly
>>>> looking revealed nothing exciting. After this confluence page
>>>> stabilizes a bit we can probably knock out some JIRAs and such.
>>>>
>>>> [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
>>>>
>>>> On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <ot...@gmail.com>
>>>> wrote:
>>>>>
>>>>> I find myself wishing I had a list of all the jiras / issues that
>> have
>>>>> been put off for a 2.0 release because they required some change or
>>>> another
>>>>> :(
>>>>>
>>>>> From: Joe Witt <jo...@gmail.com> <jo...@gmail.com>
>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
>> dev@nifi.apache.org>
>>>>> Date: July 27, 2021 at 12:30:35
>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
>>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
>>>>>
>>>>> A few thoughts:
>>>>>
>>>>> 1. I would love to see deprecation notices show up in the UI in
>>>>> various ways to help motivate users to move off things to more
>>>>> supportable things. That is not a prerequisite for anything happening
>>>>> however. Just a good feature/nice thing to do for users when someone
>>>>> is able to tackle it.
>>>>>
>>>>> 2. The decision to deprecate something and to further remove it need
>>>>> not mean there is a superior solution available. If that thing itself
>>>>> isn't getting the love/attention it needs to be
>>>>> maintained/supported/healthy going forward that alone is enough to
>>>>> remove it. That might well be the case with PostHTTP [1] and for
>>>>> comparison you can see how much effort has gone into InvokeHTTP [2].
>>>>>
>>>>> 3. When discussing a 2.0 release each thing we add as a 'must do' the
>>>>> further away from reality such a release will become. We'll have to
>>>>> get very specific about 'musts' vs 'wants'.
>>>>>
>>>>> [1]
>>>>>
>>>>
>>>
>> https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
>>>>> [2]
>>>>>
>>>>
>>>
>> https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
>>>>>
>>>>> On Tue, Jul 27, 2021 at 8:47 AM David Handermann
>>>>> <ex...@apache.org> wrote:
>>>>>>
>>>>>> Thanks Mark, providing a template or comparison statistics with
>> Java
>>>>>> versions and component configuration details would be very helpful.
>>> If
>>>> it
>>>>>> is possible to run tests using a public API or deployable service,
>>> that
>>>>>> would also help confirm potential differences.
>>>>>>
>>>>>> Showing a deprecation notice in the UI could be helpful, perhaps
>> as a
>>>>>> configurable option. NIFI-8650 describes a general Flow Analysis
>>>>>> capability, and it seems like that might be one possible way to
>>> surface
>>>>>> deprecation warnings. For something more specific to component
>>>>> deprecation,
>>>>>> it seems important to find a balance between making it obvious and
>>>> making
>>>>>> it something that ends up getting ignored.
>>>>>>
>>>>>> Regards,
>>>>>> David Handermann
>>>>>>
>>>>>> On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <ma...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>>> I'll start a new thread for PostHTTP when I get a template and/or
>>>>> detailed
>>>>>>> stats.
>>>>>>>
>>>>>>> I know the deprecation is noted in the documentation. That's a
>>>>> necessary
>>>>>>> and minimum level of notification. I was suggesting it be more
>>>> obvious
>>>>> in
>>>>>>> the UI. I think it would be beneficial to somehow be aware of the
>>>>>>> deprecation status simply by looking at the flow (perhaps even on
>>> the
>>>>>>> summary pages too), and without having to open the documentation
>>> for
>>>>> every
>>>>>>> processor to confirm whether or not a component is marked as
>>>>> deprecated.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Mark
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
>>>>>>> exceptionfactory@apache.org> wrote:
>>>>>>>
>>>>>>>> Mark,
>>>>>>>>
>>>>>>>> Thanks for the feedback. It may be better to start a separate
>>>> thread
>>>>> on
>>>>>>>> PostHTTP, but can you provide an example flow demonstrating the
>>>>>>> performance
>>>>>>>> differences between PostHTTP and InvokeHTTP?
>>>>>>>>
>>>>>>>> PostHTTP uses the Apache HttpComponents library, whereas
>>> InvokeHTTP
>>>>> uses
>>>>>>>> OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates
>> for
>>>>> OkHttp,
>>>>>>>> so it would be important to test with the most recent version.
>> It
>>>> is
>>>>> also
>>>>>>>> worth noting that test classes for PostHTTP are now integration
>>>> tests
>>>>> as
>>>>>>>> opposed to unit tests, which means they are not executed as
>> part
>>> of
>>>>> the
>>>>>>>> automated builds.
>>>>>>>>
>>>>>>>> The NiFi documentation includes the contents of the
>>>> DeprecationNotice
>>>>> for
>>>>>>>> PostHTTP:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> David Handermann
>>>>>>>>
>>>>>>>> On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <
>> mark.o.bean@gmail.com
>>>>
>>>>> wrote:
>>>>>>>>
>>>>>>>>> I'm strongly in favor of reducing tech debt, and moving
>>>>> deliberately
>>>>>>> to a
>>>>>>>>> 2.0 release. I have a concern with just one processor that is
>>>>> currently
>>>>>>>>> marked as deprecated: PostHTTP. (I have not evaluated
>>>> specifically
>>>>> any
>>>>>>>>> other deprecated components; I cannot say if there are or are
>>> not
>>>>>>> similar
>>>>>>>>> issues elsewhere.) I understand the rationale for deprecating
>>>> this
>>>>>>>>> processor in that it eliminates a processor whose
>> functionality
>>>> is
>>>>>>>>> available elsewhere, specifically in the more flexible
>>>> InvokeHTTP.
>>>>>>>> However,
>>>>>>>>> in my experience and testing, PostHTTP performs transfers
>> with
>>>> far
>>>>>>>> greater
>>>>>>>>> throughput than InvokeHTTP. I would not be in favor of
>> removing
>>>>>>> PostHTTP
>>>>>>>>> unless/until InvokeHTTP is refactored to increase its
>>> throughput
>>>>>>>>> capability.
>>>>>>>>>
>>>>>>>>> Has anyone else continued to use PostHTTP over InvokeHTTP for
>>>>> similar
>>>>>>>>> reasons? Or, is there a performance-related configuration for
>>>>>>> InvokeHTTP
>>>>>>>> I
>>>>>>>>> may have missed?
>>>>>>>>>
>>>>>>>>> Also, in order to help facilitate a smooth transition to 2.0
>>>> from a
>>>>>>> user
>>>>>>>>> perspective, would it be advisable to add some sort of visual
>>>>> indicator
>>>>>>>> in
>>>>>>>>> the UI for components that are currently annotated as
>>>> @Deprecated?
>>>>> This
>>>>>>>>> might help users proactively modify their flow prior to a
>>> release
>>>>> that
>>>>>>>>> would otherwise break it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <
>> bbende@gmail.com>
>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> With the merging of MiNiFi and registry into the NiFi repo,
>>>> we've
>>>>>>>>>> actually gone more towards a mono-repo where everything is
>>>>> released
>>>>>>>>>> together. Eventually it would still be nice to have a
>> smaller
>>>>> base
>>>>>>>>>> distribution containing just the framework and standard
>> NARs,
>>>> but
>>>>> it
>>>>>>>>>> is hard to tackle that until we provide a way for users to
>>>> easily
>>>>>>>>>> obtain the NARs in some other way.
>>>>>>>>>>
>>>>>>>>>> On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <
>>>>> edward.armes@gmail.com
>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Given the major version number shift and the spliting up
>> of
>>>>>>>> processors
>>>>>>>>>> into
>>>>>>>>>>> multiple NAR's. I'd like to suggest that we start
>>>> individually
>>>>>>>>> versioning
>>>>>>>>>>> individual NARs/ bundles.
>>>>>>>>>>>
>>>>>>>>>>> I can see this bringing a large number of benifits
>>> including
>>>>> making
>>>>>>>>> Nifi
>>>>>>>>>>> more deployable with things RPM, but also potentially
>>>> allowing
>>>>>>> those
>>>>>>>>> that
>>>>>>>>>>> have to deploy managed Nifi instances easier to upgrade.
>>>>>>>>>>>
>>>>>>>>>>> Edward
>>>>>>>>>>>
>>>>>>>>>>> On Mon, 26 Jul 2021, 20:42 Otto Fowler, <
>>>> ottobackwards@gmail.com>
>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> The issue with updating the aws sdk is if it breaks any
>>> one
>>>>> of
>>>>>>> the
>>>>>>>>>>>> processors.
>>>>>>>>>>>> the Web Gateway API invoke processor for example is not
>>>> using
>>>>> a
>>>>>>>> high
>>>>>>>>>> level
>>>>>>>>>>>> purpose build client and may break.
>>>>>>>>>>>>
>>>>>>>>>>>> If we change the aws version, we need to coordinate in
>>>> such a
>>>>> way
>>>>>>>>> that
>>>>>>>>>> they
>>>>>>>>>>>> all
>>>>>>>>>>>> can come along reasonably.
>>>>>>>>>>>> IE: what happens if 1 or 2 break but the rest or OK?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> From: David Handermann <ex...@apache.org>
>>>>>>>>>>>> <ex...@apache.org>
>>>>>>>>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
>>>>>>>>> dev@nifi.apache.org>
>>>>>>>>>>>> Date: July 26, 2021 at 09:33:42
>>>>>>>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
>>>>>>> dev@nifi.apache.org
>>>>>>>>>
>>>>>>>>>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
>>>>>>>>>>>>
>>>>>>>>>>>> Chris,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the reply and recommendations. It seems like
>>>> some
>>>>> of
>>>>>>> the
>>>>>>>>>> work to
>>>>>>>>>>>> reorganize the module structure could be done outside
>> of
>>> a
>>>>> major
>>>>>>>>>> release,
>>>>>>>>>>>> but it would be great to target any breaking changes
>> for
>>>> 2.0.
>>>>>>>>> Perhaps a
>>>>>>>>>>>> separate feature proposal on module restructuring, with
>>> the
>>>>> goal
>>>>>>> of
>>>>>>>>>>>> supporting optimized builds, would be a helpful way to
>>> move
>>>>> that
>>>>>>>> part
>>>>>>>>>> of
>>>>>>>>>>>> the discussion forward.
>>>>>>>>>>>>
>>>>>>>>>>>> Regarding updating AWS SDK to version 2, it seems like
>>> that
>>>>> might
>>>>>>>> be
>>>>>>>>>>>> possible now. I haven't taken a close look at the
>>>> referencing
>>>>>>>>>> components,
>>>>>>>>>>>> so I'm not sure about the level of effort involved.
>> Minor
>>>>> NiFi
>>>>>>>>> version
>>>>>>>>>>>> updates have incorporated major new versions of
>>>> dependencies.
>>>>> For
>>>>>>>>>> example,
>>>>>>>>>>>> NiFi 1.14 included an upgrade from Spring Framework 4
>> to
>>> 5.
>>>>> On
>>>>>>> the
>>>>>>>>> one
>>>>>>>>>>>> hand, including the AWS SDK update as part of a major
>>>> release
>>>>>>> seems
>>>>>>>>>>>> helpful, but unless there are changes that break
>> existing
>>>>>>> component
>>>>>>>>>>>> properties, upgrading the AWS SDK could be worked
>>>>> independently.
>>>>>>>>>> Others may
>>>>>>>>>>>> have more insight into particular usage of that
>> library.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> David Handermann
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
>>>>>>>>>>>> <ch...@naimuri.com.invalid> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Might be worth considering refactoring the build as
>>> part
>>>> of
>>>>>>> this
>>>>>>>>> work
>>>>>>>>>>>> too,
>>>>>>>>>>>>> e.g. only building the bits of the repo affected by a
>>>>> commit,
>>>>>>>> etc.
>>>>>>>>> -
>>>>>>>>>>>>> discussed briefly in previous threads but don't think
>>> any
>>>>>>> changes
>>>>>>>>>> made
>>>>>>>>>>>> yet.
>>>>>>>>>>>>> If NARs/components are likely to be split up and
>>>> refactored
>>>>>>> then
>>>>>>>>> such
>>>>>>>>>>>> work
>>>>>>>>>>>>> around the build probably makes sense to consider.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've a couple of PRs open that include updates to
>>>>> Elasticsearch
>>>>>>>>>> versions
>>>>>>>>>>>>> already, although I stopped at 7.10.2 (after which
>>>> Elastic
>>>>>>>> changed
>>>>>>>>>>>> licence)
>>>>>>>>>>>>> in case there were licence concerns. But more work
>> can
>>> be
>>>>> done
>>>>>>> to
>>>>>>>>>> tidy up
>>>>>>>>>>>>> the processors, absolutely.
>>>>>>>>>>>>>
>>>>>>>>>>>>> AWS libraries to v2 would seem a sensible move and a
>>>>> refactor
>>>>>>> of
>>>>>>>>>> those
>>>>>>>>>>>>> processors as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Chris Sampson
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, 24 Jul 2021, 17:47 David Handermann, <
>>>>>>>>>>>> exceptionfactory@apache.org>
>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for pointing out the standard NAR bundles,
>>> Mark.
>>>>> There
>>>>>>>>> are a
>>>>>>>>>>>>> number
>>>>>>>>>>>>>> of components in the standard NAR bundles with
>>>> particular
>>>>>>>>>> dependencies
>>>>>>>>>>>>> that
>>>>>>>>>>>>>> would make more sense in separate NARs.
>> Reorganizing
>>>> the
>>>>>>>> standard
>>>>>>>>>> NAR
>>>>>>>>>>>> to
>>>>>>>>>>>>>> components with limited dependencies and wide
>>>>> applicability
>>>>>>>> would
>>>>>>>>>>>>>> definitely help with future maintenance.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> David Handermann
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
>>>>>>>>> markap14@hotmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There’s also some code that exists in order to
>>>> maintain
>>>>>>>>> backward
>>>>>>>>>>>>>>> compatibility in the repositories. I would very
>>> much
>>>>> like
>>>>>>> the
>>>>>>>>>>>>>> repositories
>>>>>>>>>>>>>>> to contain no unnecessary code. And swap file
>>> format
>>>>>>> supports
>>>>>>>>>> really
>>>>>>>>>>>>> old
>>>>>>>>>>>>>>> formats. And the old impls of the repositories
>>>>> themselves,
>>>>>>>> like
>>>>>>>>>>>>>>> PersistentProvRepo instead of WriteAheadProv
>> Repo,
>>>> etc.
>>>>>>> Lots
>>>>>>>> of
>>>>>>>>>> stuff
>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>> that can be removed. And some methods in
>>>> ProcessSession
>>>>>>> that
>>>>>>>>> are
>>>>>>>>>>>> never
>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>> by any processor in the codebase but exists in
>> the
>>>>> public
>>>>>>> API
>>>>>>>>> so
>>>>>>>>>>>> can’t
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>> removed till 2.0.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think his is also a great time to clean up the
>>>>> “standard
>>>>>>>>> nar.”
>>>>>>>>>> At
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> point, it’s something like 70 MB. And many of the
>>>>>>> components
>>>>>>>>>> there
>>>>>>>>>>>> are
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>> really “standard” - things like connecting to
>> FTP &
>>>>> SFTP
>>>>>>>>>> servers, XML
>>>>>>>>>>>>>>> processing, Jolt transform, etc. could
>> potentially
>>> be
>>>>> moved
>>>>>>>>> into
>>>>>>>>>>>> other
>>>>>>>>>>>>>>> nars. The
>>>>> nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
>>>>>>> is
>>>>>>>>>> 6.9 MB
>>>>>>>>>>>> is
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>> necessary for stateless or minifi java. Lots of
>>>> things
>>>>>>>> probably
>>>>>>>>>> to
>>>>>>>>>>>>>>> reconsider within the standard nar.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I definitely think this is a reasonable approach,
>>> to
>>>>> allow
>>>>>>>> for
>>>>>>>>> a
>>>>>>>>>> 2.0
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> is not a huge feature release but allows the
>>> project
>>>> to
>>>>> be
>>>>>>>>>> simpler
>>>>>>>>>>>> and
>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>> nimble.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> -Mark
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
>>>>>>>>>> mikerthomsen@gmail.com
>>>>>>>>>>>>>> <mailto:
>>>>>>>>>>>>>>> mikerthomsen@gmail.com>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Russell,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> AFAICT from looking at Elastic's repos, the low
>>> level
>>>>> REST
>>>>>>>>>> client is
>>>>>>>>>>>>>>> still fine.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Our Elasticsearch support is spread over two NARs
>>> at
>>>>>>> present.
>>>>>>>>> One
>>>>>>>>>>>> uses
>>>>>>>>>>>>>>> OkHttp and the other uses that low level Elastic
>>> REST
>>>>>>> client.
>>>>>>>>>>>>>>> Therefore, I think we're fine on licensing for
>> the
>>>>> moment.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
>>>>>>>>>>>> russ@windofkeltia.com
>>>>>>>>>>>>>>> <ma...@windofkeltia.com>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Bringing up Elastic also reminds me that the
>>> Elastic
>>>>>>>> framework
>>>>>>>>>> has
>>>>>>>>>>>> just
>>>>>>>>>>>>>>> recently transitioned out of Open Source, so to
>>>>> acknowledge
>>>>>>>>> that,
>>>>>>>>>>>> maybe
>>>>>>>>>>>>>>> some effort toward OpenSearch--I say this not
>>>>> understanding
>>>>>>>>>> exactly
>>>>>>>>>>>> how
>>>>>>>>>>>>>>> this sort of thing is considered in a
>> large-scale,
>>>>>>>> world-class
>>>>>>>>>>>> software
>>>>>>>>>>>>>>> project like Apache NiFi. (I'm not a contributor,
>>>> just
>>>>> a
>>>>>>>>> grateful
>>>>>>>>>>>>>>> consumer.)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Russ
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 7/23/21 10:28 AM, Matt Burgess wrote:
>>>>>>>>>>>>>>> Along with the itemized list for ancient
>> components
>>>> we
>>>>>>> should
>>>>>>>>>> look at
>>>>>>>>>>>>>>> updating versions of drivers, SDKs, etc. for
>>> external
>>>>>>> systems
>>>>>>>>>> such as
>>>>>>>>>>>>>>> Elasticsearch, Cassandra, etc. There may be
>>> breaking
>>>>>>> changes
>>>>>>>>> but
>>>>>>>>>> 2.0
>>>>>>>>>>>>>>> is probably the right time to get things up to
>> date
>>>> to
>>>>> make
>>>>>>>>> them
>>>>>>>>>> more
>>>>>>>>>>>>>>> useful to more people.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
>>>>>>>>>> thenatog@gmail.com
>>>>>>>>>>>>>> <mailto:
>>>>>>>>>>>>>>> thenatog@gmail.com>> wrote:
>>>>>>>>>>>>>>> I'm a +1 for removing pretty much all of this
>>> stuff.
>>>>> There
>>>>>>>> are
>>>>>>>>>>>> security
>>>>>>>>>>>>>>> implications to keeping old dependencies around,
>> so
>>>> the
>>>>>>> more
>>>>>>>>> old
>>>>>>>>>> code
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>> can remove the better. I agree that eventually we
>>>> need
>>>>> to
>>>>>>>> move
>>>>>>>>> to
>>>>>>>>>>>>>>> supporting only Java 11+, and as our next release
>>>> will
>>>>>>>> probably
>>>>>>>>>> be
>>>>>>>>>>>>> about
>>>>>>>>>>>>>> 4
>>>>>>>>>>>>>>> - 6 months from now that doesn't seem too soon.
>> We
>>>>> could
>>>>>>>>>> potentially
>>>>>>>>>>>>>> break
>>>>>>>>>>>>>>> this in two and remove the deprecated processors
>>> and
>>>>> leave
>>>>>>>> 1.x
>>>>>>>>> on
>>>>>>>>>>>> Java
>>>>>>>>>>>>> 8,
>>>>>>>>>>>>>>> and finally start on 2.x which would support only
>>>> Java
>>>>> 11.
>>>>>>>> I'm
>>>>>>>>>> unsure
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> what implications changing the date and time
>>> handling
>>>>> would
>>>>>>>>> have
>>>>>>>>>> -
>>>>>>>>>>>> for
>>>>>>>>>>>>>>> running systems that use long term historical
>> logs,
>>>>>>>> unexpected
>>>>>>>>>>>> impacts
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> time logging could be a problem.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As Joe says I think feature work will have to be
>>>>> dedicated
>>>>>>> to
>>>>>>>>>> 2.x and
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>> could support 1.x for security fixes for some
>>> period
>>>> of
>>>>>>> time.
>>>>>>>>> 2.x
>>>>>>>>>>>> seems
>>>>>>>>>>>>>>> like a gargantuan task but it's probably time to
>>> get
>>>>>>> started.
>>>>>>>>> Not
>>>>>>>>>>>> sure
>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>> we handle all open PRs and the transition between
>>> 1.x
>>>>> and
>>>>>>>> 2.x.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
>>>>>>>> joe.witt@gmail.com
>>>>>>>>>>>> <mailto:
>>>>>>>>>>>>>>> joe.witt@gmail.com>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You're right we have to be careful and you're
>> right
>>>>> there
>>>>>>> are
>>>>>>>>>> still
>>>>>>>>>>>>>>> significant Java 8 users out there. But we also
>>> have
>>>> to
>>>>> be
>>>>>>>>>> careful
>>>>>>>>>>>>>>> about security and sustainability of the
>> codebase.
>>> If
>>>>> we
>>>>>>> had
>>>>>>>>>> talked
>>>>>>>>>>>>>>> about this last year when that article came out
>> I'd
>>>>> have
>>>>>>>> agreed
>>>>>>>>>> it is
>>>>>>>>>>>>>>> too early. Interestingly that link seems to get
>>>> updated
>>>>>>> and I
>>>>>>>>>> tried
>>>>>>>>>>>>>>> [1] and found more recent data (not sure how
>>> recent).
>>>>>>> Anyway
>>>>>>>> it
>>>>>>>>>>>>>>> suggests Java 8 is still the top dog but we see
>>> good
>>>>> growth
>>>>>>>> on
>>>>>>>>>> 11. In
>>>>>>>>>>>>>>> my $dayjob this aligns to what I'm seeing too.
>>>>> Customers
>>>>>>>> didn't
>>>>>>>>>> seem
>>>>>>>>>>>>>>> to care about Java 11 until later half last year
>>> and
>>>>> now
>>>>>>>>>> suddenly it
>>>>>>>>>>>>>>> is all over the place.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think once we put out a NiFi 2.0 release we'd
>> see
>>>>> rapid
>>>>>>>>>> decrease in
>>>>>>>>>>>>>>> work on the 1.x line just being blunt. We did
>> this
>>>> many
>>>>>>> years
>>>>>>>>> ago
>>>>>>>>>>>>>>> with 0.x to 1.x and we stood behind 0.x for a
>> while
>>>>> (maybe
>>>>>>> a
>>>>>>>>>> year or
>>>>>>>>>>>>>>> so) but it was purely bug fix/security related
>>> bits.
>>>> We
>>>>>>> would
>>>>>>>>>> need to
>>>>>>>>>>>>>>> do something similar. But feature work would
>> almost
>>>>>>> certainly
>>>>>>>>> go
>>>>>>>>>> to
>>>>>>>>>>>>>>> the 2.x line. Maybe there are other workable
>> models
>>>> but
>>>>> my
>>>>>>>>>> instinct
>>>>>>>>>>>>>>> suggests this is likely to follow a similar path.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ...anyway I agree it isn't that easy of a call to
>>>> dump
>>>>> Java
>>>>>>>> 8.
>>>>>>>>> We
>>>>>>>>>>>>>>> need to make the call in both the interests of
>> the
>>>> user
>>>>>>> base
>>>>>>>>> and
>>>>>>>>>> the
>>>>>>>>>>>>>>> contributor base of the community.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>> https://www.jetbrains.com/lp/devecosystem-2021/java/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> Joe
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
>>>>>>> joe.witt@gmail.com
>>>>>>>>>> <mailto:
>>>>>>>>>>>>>>> joe.witt@gmail.com>> wrote:
>>>>>>>>>>>>>>> Russ
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yeah the flow registry is a key part of it. But
>>> also
>>>>> now
>>>>>>> you
>>>>>>>>> can
>>>>>>>>>>>>>>> download the flow definition in JSON (upload i
>>> think
>>>> is
>>>>>>> there
>>>>>>>>> now
>>>>>>>>>>>>>>> too). Templates offered a series of challenges
>> such
>>>> as
>>>>> we
>>>>>>>> store
>>>>>>>>>> them
>>>>>>>>>>>>>>> in the flow definition which has made flows
>> massive
>>>> in
>>>>> an
>>>>>>>>>> unintended
>>>>>>>>>>>>>>> way which isn't fun for cluster behavior.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We have a couple cases where we headed down a
>>>>> particular
>>>>>>>>> concept
>>>>>>>>>> and
>>>>>>>>>>>>>>> came up with better approaches later. We need to
>>>>> reconcile
>>>>>>>>> these
>>>>>>>>>> with
>>>>>>>>>>>>>>> the benefit of hindsight, and while being careful
>>> to
>>>> be
>>>>> not
>>>>>>>>>> overly
>>>>>>>>>>>>>>> disruptive to existing users, to reduce the
>>>>>>>>> codebase/maintenance
>>>>>>>>>>>>>>> burden and allow continued evolution of the
>>> project.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
>>>>>>>>>>>> russ@windofkeltia.com
>>>>>>>>>>>>>>> <ma...@windofkeltia.com>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> Joe,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I apologize for the off-topic intrusion, but what
>>>>> replaces
>>>>>>>>>> templates?
>>>>>>>>>>>>>>> The Registry? Templates rocked and we have used
>>> them
>>>>> since
>>>>>>>>> 0.5.x.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Russ
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 7/23/21 8:31 AM, Joe Witt wrote:
>>>>>>>>>>>>>>> David,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think this is a highly reasonable approach and
>>>> such a
>>>>>>> focus
>>>>>>>>>> will
>>>>>>>>>>>>>>> greatly help make a 2.0 release far more
>>> approachable
>>>>> to
>>>>>>>> knock
>>>>>>>>>> out.
>>>>>>>>>>>>>>> Not only that but tech debt reduction would help
>>> make
>>>>> work
>>>>>>>>>> towards
>>>>>>>>>>>>>>> major features we'd think about in a 'major
>>> release'
>>>>> sense
>>>>>>>> more
>>>>>>>>>>>>>>> approachable.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We should remove all deprecated things (as well
>> as
>>>>> verify
>>>>>>> we
>>>>>>>>>> have the
>>>>>>>>>>>>>>> right list). We should remove/consider removal of
>>>>>>> deprecated
>>>>>>>>>>>>>>> concepts
>>>>>>>>>>>>>>> like templates. We should consider whether we can
>>>>> resolve
>>>>>>> the
>>>>>>>>>>>>>>> various
>>>>>>>>>>>>>>> ways we've handled what are now parameters down
>> to
>>>> one
>>>>>>> clean
>>>>>>>>>>>>>>> approach.
>>>>>>>>>>>>>>> We should remove options in the nifi.properties
>>> which
>>>>> turn
>>>>>>>> out
>>>>>>>>> to
>>>>>>>>>>>>>>> never be used quite right (if there are). There
>> is
>>>>> quite a
>>>>>>>> bit
>>>>>>>>> we
>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>> do purely in the name of tech debt reduction.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Lots to consider here but I think this is the
>> right
>>>>>>>> discussion.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Than ks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
>>>>>>>> bbende@gmail.com
>>>>>>>>>>>> <mailto:
>>>>>>>>>>>>>>> bbende@gmail.com>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> I'm a +1 for this... Not sure if this falls under
>>>>> "Removing
>>>>>>>>>>>>>>> Deprecated
>>>>>>>>>>>>>>> Components", but I think we should also look at
>>>>> anything
>>>>>>> that
>>>>>>>>> has
>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>> marked as deprecated throughout the code base as
>> a
>>>>>>> candidate
>>>>>>>>> for
>>>>>>>>>>>>>>> removal. There are quite a few classes, methods,
>>>>>>> properties,
>>>>>>>>> etc
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> have been waiting for a chance to be removed.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
>>>>>>>>>>>>>>> <exceptionfactory@apache.org<mailto:
>>>>>>>>> exceptionfactory@apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> Team,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> With all of the excellent work that many have
>>>>> contributed
>>>>>>> to
>>>>>>>>> NiFi
>>>>>>>>>>>>>>> over the
>>>>>>>>>>>>>>> years, the code base has also accumulated some
>>> amount
>>>>> of
>>>>>>>>>> technical
>>>>>>>>>>>>>>> debt. A
>>>>>>>>>>>>>>> handful of components have been marked as
>>> deprecated,
>>>>> and
>>>>>>>> some
>>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>> remain in the code base to support integration
>> with
>>>> old
>>>>>>>>> versions
>>>>>>>>>>>>>>> of various
>>>>>>>>>>>>>>> products. Following the principles of semantic
>>>>> versioning,
>>>>>>>>>>>>>>> introducing a
>>>>>>>>>>>>>>> major release would provide the opportunity to
>>> remove
>>>>> these
>>>>>>>>>>>>>>> deprecated and
>>>>>>>>>>>>>>> unsupported components.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Rather than focusing the next major release on
>> new
>>>>>>> features,
>>>>>>>>> what
>>>>>>>>>>>>>>> do you
>>>>>>>>>>>>>>> think about focusing on technical debt removal?
>>> This
>>>>>>> approach
>>>>>>>>>>>>>>> would not
>>>>>>>>>>>>>>> make for the most interesting release, but it
>>>> provides
>>>>> the
>>>>>>>>>>>>>>> opportunity to
>>>>>>>>>>>>>>> clean up elements that involve breaking changes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Focusing on technical debt, at least three
>> primary
>>>>> goals
>>>>>>> come
>>>>>>>>> to
>>>>>>>>>>>>>>> mind for
>>>>>>>>>>>>>>> the next major release:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1. Removal of deprecated and unmaintained
>>> components
>>>>>>>>>>>>>>> 2. Require Java 11 as the minimum supported
>> version
>>>>>>>>>>>>>>> 3. Transition internal date and time handling to
>>> JSR
>>>>> 310
>>>>>>>>>> java.time
>>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *Removing Deprecated Components*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Removing support for older and deprecated
>>> components
>>>>>>>> provides a
>>>>>>>>>>>>>>> great
>>>>>>>>>>>>>>> opportunity to improve the overall security
>> posture
>>>>> when it
>>>>>>>>> comes
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> maintaining dependencies. The OWASP dependency
>>> plugin
>>>>>>> report
>>>>>>>>>>>>>>> currently
>>>>>>>>>>>>>>> generates 50 MB of HTML for questionable
>>>> dependencies,
>>>>> many
>>>>>>>> of
>>>>>>>>>>>>>>> which are
>>>>>>>>>>>>>>> related to old versions of various libraries.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As a starting point, here are a handful of
>>> components
>>>>> and
>>>>>>>>>>>>>>> extension modules
>>>>>>>>>>>>>>> that could be targeted for removal in a major
>>>> version:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - PostHTTP and GetHTTP
>>>>>>>>>>>>>>> - ListenLumberjack and the entire
>>>>> nifi-lumberjack-bundle
>>>>>>>>>>>>>>> - ListenBeats and the entire nifi-beats-bundle
>>>>>>>>>>>>>>> - Elasticsearch 5 components
>>>>>>>>>>>>>>> - Hive 1 and 2 components
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *Requiring Java 11*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Java 8 is now over seven years old, and NiFi has
>>>>> supported
>>>>>>>>>> general
>>>>>>>>>>>>>>> compatibility with Java 11 for several years.
>> NiFi
>>>>> 1.14.0
>>>>>>>>>>>>>>> incorporated
>>>>>>>>>>>>>>> internal improvements specifically related to TLS
>>>> 1.3,
>>>>>>> which
>>>>>>>>>>>>>>> allowed
>>>>>>>>>>>>>>> closing out the long-running Java 11
>> compatibility
>>>> epic
>>>>>>>>>> NIFI-5174.
>>>>>>>>>>>>>>> Making
>>>>>>>>>>>>>>> Java 11 the minimum required version provides the
>>>>>>> opportunity
>>>>>>>>> to
>>>>>>>>>>>>>>> address
>>>>>>>>>>>>>>> any lingering edge cases and put NiFi in a better
>>>>> position
>>>>>>> to
>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>> current Java versions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *JSR 310 for Date and Time Handling*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Without making the scope too broad, transitioning
>>>>> internal
>>>>>>>> date
>>>>>>>>>>>>>>> and time
>>>>>>>>>>>>>>> handling to use DateTimeFormatter instead of
>>>>>>> SimpleDateFormat
>>>>>>>>>>>>>>> would provide
>>>>>>>>>>>>>>> a number of advantages. The Java Time components
>>>>> provide
>>>>>>> much
>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>> clarity when it comes to handling localized date
>>> and
>>>>> time
>>>>>>>>>>>>>>> representations,
>>>>>>>>>>>>>>> and also avoid the inherent confusion of
>>>> java.sql.Date
>>>>>>>>> extending
>>>>>>>>>>>>>>> java.util.Date. Many internal components,
>>>> specifically
>>>>>>>>>>>>>>> Record-oriented
>>>>>>>>>>>>>>> processors and services, rely on date parsing,
>>>> leading
>>>>> to
>>>>>>>>>>>>>>> confusion and
>>>>>>>>>>>>>>> various workarounds. The pattern formats of
>>>>>>> SimpleDateFormat
>>>>>>>>> and
>>>>>>>>>>>>>>> DateTimeFormatter are very similar, but there
>> are a
>>>> few
>>>>>>>> subtle
>>>>>>>>>>>>>>> differences.
>>>>>>>>>>>>>>> Making this transition would provide a much
>> better
>>>>>>> foundation
>>>>>>>>>>>>>>> going forward.
>>>>>>>>>>>>>>> *Conclusion*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for giving this proposal some
>> consideration.
>>>>> Many of
>>>>>>>> you
>>>>>>>>>>>>>>> have been
>>>>>>>>>>>>>>> developing NiFi for years and I look forward to
>>> your
>>>>>>>> feedback.
>>>>>>>>> I
>>>>>>>>>>>>>>> would be
>>>>>>>>>>>>>>> glad to put together a more formalized
>>> recommendation
>>>>> on
>>>>>>>>>>>>>>> Confluence and
>>>>>>>>>>>>>>> write up Jira epics if this general approach
>> sounds
>>>>>>> agreeable
>>>>>>>>> to
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> community.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> David Handermann
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>>>
>>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by David Handermann <ex...@gmail.com>.
Thanks for the continued discussion around what can or should be removed.
Having a clear migration path from version 1 to version 2 will be very
important for supporting current deployments.
Following the example of some other projects, one way to consider the
upgrade is from the point of view of deprecation warnings. If implemented
correctly, an installation of NiFi running the latest minor release of
version 1, and showing no deprecation warnings, should be upgradeable to
version 2 without any changes. Making this work involves accurately tagging
components and methods with deprecation indicators, and providing a clear
way to observe deprecation warnings. With the current version containing
both deprecated components and deprecated methods in various classes, this
would involve some work to get right. A simple approach might be a named
logger for deprecation warnings, which would write a separate log file. A
more advanced approach might involve a tool to analyze the system and flow
configurations. Either way, it seems that some additional work in a minor
release version is necessary before considering a major release version.
One other point on compatibility: as long as the core component API
definition remains backwards compatible, it should be possible to run older
components. This provides a transition option for customers using
components such as PostHTTP, or any others that are not actively
maintained. At some point it may become necessary to break compatibility at
that level, but at least for version 2, maintaining component API
compatibility is key.
Regards,
David Handermann
On Sun, Aug 1, 2021 at 10:23 AM Mark Bean <ma...@gmail.com> wrote:
> I created a JIRA ticket to investigate and improve the performance of
> InvokeHTTP. It includes a flow definition for benchmarking the performance
> of both PostHTTP and InvokeHTTP.
>
> https://issues.apache.org/jira/browse/NIFI-8968
>
> Thanks,
> Mark
>
> On Fri, Jul 30, 2021 at 6:41 PM Adam Taft <ad...@adamtaft.com> wrote:
>
> > I'm not seeing the side thread that was going to discuss deprecation of
> > PostHTTP. Has that thread started and I just don't see it?
> >
> > One (significant?) concern with PostHTTP is the smooth integration of
> > NiFi-to-NiFi communication that is very transparently enabled with the
> > ListenHTTP and PostHTTP processors. There's some special logic in there
> for
> > handling flowfiles that InvokeHTTP doesn't really (nor should really)
> have.
> >
> > I know of several (large) NiFi installations that rely on the PostHTTP /
> > ListenHTTP combination. It has enabled NiFi to NiFi communication for
> folks
> > reluctant or unable to enable site-to-site type configuration.
> >
> > Honestly, I don't know why we'd want to "deprecate" any processor, as
> > opposed to just moving it to a new location. If these processors can be
> > ported and maintained to whatever the 2.0 API looks like, there's
> possibly
> > little harm keeping them around.
> >
> > And by the way, what happened to the "marketplace" concept? Is this being
> > considered for 2.0 as well? Because relocating the deprecated processors
> > to an external nar might be the best solution. Losing PostHTTP entirely I
> > think would be a mistake, but I'd conceptually support relocating it.
> >
> > Thanks,
> >
> > /Adam
> >
> > On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <jo...@gmail.com> wrote:
> >
> > > Looks like we just need to knock out 5 JIRAs :) [1]
> > >
> > > I felt like we had a label folks were using at one point but quickly
> > > looking revealed nothing exciting. After this confluence page
> > > stabilizes a bit we can probably knock out some JIRAs and such.
> > >
> > > [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
> > >
> > > On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <ot...@gmail.com>
> > > wrote:
> > > >
> > > > I find myself wishing I had a list of all the jiras / issues that
> have
> > > > been put off for a 2.0 release because they required some change or
> > > another
> > > > :(
> > > >
> > > > From: Joe Witt <jo...@gmail.com> <jo...@gmail.com>
> > > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> dev@nifi.apache.org>
> > > > Date: July 27, 2021 at 12:30:35
> > > > To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > >
> > > > A few thoughts:
> > > >
> > > > 1. I would love to see deprecation notices show up in the UI in
> > > > various ways to help motivate users to move off things to more
> > > > supportable things. That is not a prerequisite for anything happening
> > > > however. Just a good feature/nice thing to do for users when someone
> > > > is able to tackle it.
> > > >
> > > > 2. The decision to deprecate something and to further remove it need
> > > > not mean there is a superior solution available. If that thing itself
> > > > isn't getting the love/attention it needs to be
> > > > maintained/supported/healthy going forward that alone is enough to
> > > > remove it. That might well be the case with PostHTTP [1] and for
> > > > comparison you can see how much effort has gone into InvokeHTTP [2].
> > > >
> > > > 3. When discussing a 2.0 release each thing we add as a 'must do' the
> > > > further away from reality such a release will become. We'll have to
> > > > get very specific about 'musts' vs 'wants'.
> > > >
> > > > [1]
> > > >
> > >
> >
> https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
> > > > [2]
> > > >
> > >
> >
> https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
> > > >
> > > > On Tue, Jul 27, 2021 at 8:47 AM David Handermann
> > > > <ex...@apache.org> wrote:
> > > > >
> > > > > Thanks Mark, providing a template or comparison statistics with
> Java
> > > > > versions and component configuration details would be very helpful.
> > If
> > > it
> > > > > is possible to run tests using a public API or deployable service,
> > that
> > > > > would also help confirm potential differences.
> > > > >
> > > > > Showing a deprecation notice in the UI could be helpful, perhaps
> as a
> > > > > configurable option. NIFI-8650 describes a general Flow Analysis
> > > > > capability, and it seems like that might be one possible way to
> > surface
> > > > > deprecation warnings. For something more specific to component
> > > > deprecation,
> > > > > it seems important to find a balance between making it obvious and
> > > making
> > > > > it something that ends up getting ignored.
> > > > >
> > > > > Regards,
> > > > > David Handermann
> > > > >
> > > > > On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <ma...@gmail.com>
> > > wrote:
> > > > >
> > > > > > I'll start a new thread for PostHTTP when I get a template and/or
> > > > detailed
> > > > > > stats.
> > > > > >
> > > > > > I know the deprecation is noted in the documentation. That's a
> > > > necessary
> > > > > > and minimum level of notification. I was suggesting it be more
> > > obvious
> > > > in
> > > > > > the UI. I think it would be beneficial to somehow be aware of the
> > > > > > deprecation status simply by looking at the flow (perhaps even on
> > the
> > > > > > summary pages too), and without having to open the documentation
> > for
> > > > every
> > > > > > processor to confirm whether or not a component is marked as
> > > > deprecated.
> > > > > >
> > > > > > Thanks,
> > > > > > Mark
> > > > > >
> > > > > >
> > > > > > On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
> > > > > > exceptionfactory@apache.org> wrote:
> > > > > >
> > > > > > > Mark,
> > > > > > >
> > > > > > > Thanks for the feedback. It may be better to start a separate
> > > thread
> > > > on
> > > > > > > PostHTTP, but can you provide an example flow demonstrating the
> > > > > > performance
> > > > > > > differences between PostHTTP and InvokeHTTP?
> > > > > > >
> > > > > > > PostHTTP uses the Apache HttpComponents library, whereas
> > InvokeHTTP
> > > > uses
> > > > > > > OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates
> for
> > > > OkHttp,
> > > > > > > so it would be important to test with the most recent version.
> It
> > > is
> > > > also
> > > > > > > worth noting that test classes for PostHTTP are now integration
> > > tests
> > > > as
> > > > > > > opposed to unit tests, which means they are not executed as
> part
> > of
> > > > the
> > > > > > > automated builds.
> > > > > > >
> > > > > > > The NiFi documentation includes the contents of the
> > > DeprecationNotice
> > > > for
> > > > > > > PostHTTP:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
> > > > > > >
> > > > > > > Regards,
> > > > > > > David Handermann
> > > > > > >
> > > > > > > On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <
> mark.o.bean@gmail.com
> > >
> > > > wrote:
> > > > > > >
> > > > > > > > I'm strongly in favor of reducing tech debt, and moving
> > > > deliberately
> > > > > > to a
> > > > > > > > 2.0 release. I have a concern with just one processor that is
> > > > currently
> > > > > > > > marked as deprecated: PostHTTP. (I have not evaluated
> > > specifically
> > > > any
> > > > > > > > other deprecated components; I cannot say if there are or are
> > not
> > > > > > similar
> > > > > > > > issues elsewhere.) I understand the rationale for deprecating
> > > this
> > > > > > > > processor in that it eliminates a processor whose
> functionality
> > > is
> > > > > > > > available elsewhere, specifically in the more flexible
> > > InvokeHTTP.
> > > > > > > However,
> > > > > > > > in my experience and testing, PostHTTP performs transfers
> with
> > > far
> > > > > > > greater
> > > > > > > > throughput than InvokeHTTP. I would not be in favor of
> removing
> > > > > > PostHTTP
> > > > > > > > unless/until InvokeHTTP is refactored to increase its
> > throughput
> > > > > > > > capability.
> > > > > > > >
> > > > > > > > Has anyone else continued to use PostHTTP over InvokeHTTP for
> > > > similar
> > > > > > > > reasons? Or, is there a performance-related configuration for
> > > > > > InvokeHTTP
> > > > > > > I
> > > > > > > > may have missed?
> > > > > > > >
> > > > > > > > Also, in order to help facilitate a smooth transition to 2.0
> > > from a
> > > > > > user
> > > > > > > > perspective, would it be advisable to add some sort of visual
> > > > indicator
> > > > > > > in
> > > > > > > > the UI for components that are currently annotated as
> > > @Deprecated?
> > > > This
> > > > > > > > might help users proactively modify their flow prior to a
> > release
> > > > that
> > > > > > > > would otherwise break it.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <
> bbende@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > With the merging of MiNiFi and registry into the NiFi repo,
> > > we've
> > > > > > > > > actually gone more towards a mono-repo where everything is
> > > > released
> > > > > > > > > together. Eventually it would still be nice to have a
> smaller
> > > > base
> > > > > > > > > distribution containing just the framework and standard
> NARs,
> > > but
> > > > it
> > > > > > > > > is hard to tackle that until we provide a way for users to
> > > easily
> > > > > > > > > obtain the NARs in some other way.
> > > > > > > > >
> > > > > > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <
> > > > edward.armes@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Given the major version number shift and the spliting up
> of
> > > > > > > processors
> > > > > > > > > into
> > > > > > > > > > multiple NAR's. I'd like to suggest that we start
> > > individually
> > > > > > > > versioning
> > > > > > > > > > individual NARs/ bundles.
> > > > > > > > > >
> > > > > > > > > > I can see this bringing a large number of benifits
> > including
> > > > making
> > > > > > > > Nifi
> > > > > > > > > > more deployable with things RPM, but also potentially
> > > allowing
> > > > > > those
> > > > > > > > that
> > > > > > > > > > have to deploy managed Nifi instances easier to upgrade.
> > > > > > > > > >
> > > > > > > > > > Edward
> > > > > > > > > >
> > > > > > > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, <
> > > ottobackwards@gmail.com>
> > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > The issue with updating the aws sdk is if it breaks any
> > one
> > > > of
> > > > > > the
> > > > > > > > > > > processors.
> > > > > > > > > > > the Web Gateway API invoke processor for example is not
> > > using
> > > > a
> > > > > > > high
> > > > > > > > > level
> > > > > > > > > > > purpose build client and may break.
> > > > > > > > > > >
> > > > > > > > > > > If we change the aws version, we need to coordinate in
> > > such a
> > > > way
> > > > > > > > that
> > > > > > > > > they
> > > > > > > > > > > all
> > > > > > > > > > > can come along reasonably.
> > > > > > > > > > > IE: what happens if 1 or 2 break but the rest or OK?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > From: David Handermann <ex...@apache.org>
> > > > > > > > > > > <ex...@apache.org>
> > > > > > > > > > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > > > > > dev@nifi.apache.org>
> > > > > > > > > > > Date: July 26, 2021 at 09:33:42
> > > > > > > > > > > To: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > > > dev@nifi.apache.org
> > > > > > > >
> > > > > > > > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > > > > > > > > >
> > > > > > > > > > > Chris,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the reply and recommendations. It seems like
> > > some
> > > > of
> > > > > > the
> > > > > > > > > work to
> > > > > > > > > > > reorganize the module structure could be done outside
> of
> > a
> > > > major
> > > > > > > > > release,
> > > > > > > > > > > but it would be great to target any breaking changes
> for
> > > 2.0.
> > > > > > > > Perhaps a
> > > > > > > > > > > separate feature proposal on module restructuring, with
> > the
> > > > goal
> > > > > > of
> > > > > > > > > > > supporting optimized builds, would be a helpful way to
> > move
> > > > that
> > > > > > > part
> > > > > > > > > of
> > > > > > > > > > > the discussion forward.
> > > > > > > > > > >
> > > > > > > > > > > Regarding updating AWS SDK to version 2, it seems like
> > that
> > > > might
> > > > > > > be
> > > > > > > > > > > possible now. I haven't taken a close look at the
> > > referencing
> > > > > > > > > components,
> > > > > > > > > > > so I'm not sure about the level of effort involved.
> Minor
> > > > NiFi
> > > > > > > > version
> > > > > > > > > > > updates have incorporated major new versions of
> > > dependencies.
> > > > For
> > > > > > > > > example,
> > > > > > > > > > > NiFi 1.14 included an upgrade from Spring Framework 4
> to
> > 5.
> > > > On
> > > > > > the
> > > > > > > > one
> > > > > > > > > > > hand, including the AWS SDK update as part of a major
> > > release
> > > > > > seems
> > > > > > > > > > > helpful, but unless there are changes that break
> existing
> > > > > > component
> > > > > > > > > > > properties, upgrading the AWS SDK could be worked
> > > > independently.
> > > > > > > > > Others may
> > > > > > > > > > > have more insight into particular usage of that
> library.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > David Handermann
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > > > > > > > > > > <ch...@naimuri.com.invalid> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Might be worth considering refactoring the build as
> > part
> > > of
> > > > > > this
> > > > > > > > work
> > > > > > > > > > > too,
> > > > > > > > > > > > e.g. only building the bits of the repo affected by a
> > > > commit,
> > > > > > > etc.
> > > > > > > > -
> > > > > > > > > > > > discussed briefly in previous threads but don't think
> > any
> > > > > > changes
> > > > > > > > > made
> > > > > > > > > > > yet.
> > > > > > > > > > > > If NARs/components are likely to be split up and
> > > refactored
> > > > > > then
> > > > > > > > such
> > > > > > > > > > > work
> > > > > > > > > > > > around the build probably makes sense to consider.
> > > > > > > > > > > >
> > > > > > > > > > > > I've a couple of PRs open that include updates to
> > > > Elasticsearch
> > > > > > > > > versions
> > > > > > > > > > > > already, although I stopped at 7.10.2 (after which
> > > Elastic
> > > > > > > changed
> > > > > > > > > > > licence)
> > > > > > > > > > > > in case there were licence concerns. But more work
> can
> > be
> > > > done
> > > > > > to
> > > > > > > > > tidy up
> > > > > > > > > > > > the processors, absolutely.
> > > > > > > > > > > >
> > > > > > > > > > > > AWS libraries to v2 would seem a sensible move and a
> > > > refactor
> > > > > > of
> > > > > > > > > those
> > > > > > > > > > > > processors as well.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > >
> > > > > > > > > > > > Chris Sampson
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > > > > > > > > > > exceptionfactory@apache.org>
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for pointing out the standard NAR bundles,
> > Mark.
> > > > There
> > > > > > > > are a
> > > > > > > > > > > > number
> > > > > > > > > > > > > of components in the standard NAR bundles with
> > > particular
> > > > > > > > > dependencies
> > > > > > > > > > > > that
> > > > > > > > > > > > > would make more sense in separate NARs.
> Reorganizing
> > > the
> > > > > > > standard
> > > > > > > > > NAR
> > > > > > > > > > > to
> > > > > > > > > > > > > components with limited dependencies and wide
> > > > applicability
> > > > > > > would
> > > > > > > > > > > > > definitely help with future maintenance.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > David Handermann
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> > > > > > > > markap14@hotmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > There’s also some code that exists in order to
> > > maintain
> > > > > > > > backward
> > > > > > > > > > > > > > compatibility in the repositories. I would very
> > much
> > > > like
> > > > > > the
> > > > > > > > > > > > > repositories
> > > > > > > > > > > > > > to contain no unnecessary code. And swap file
> > format
> > > > > > supports
> > > > > > > > > really
> > > > > > > > > > > > old
> > > > > > > > > > > > > > formats. And the old impls of the repositories
> > > > themselves,
> > > > > > > like
> > > > > > > > > > > > > > PersistentProvRepo instead of WriteAheadProv
> Repo,
> > > etc.
> > > > > > Lots
> > > > > > > of
> > > > > > > > > stuff
> > > > > > > > > > > > > there
> > > > > > > > > > > > > > that can be removed. And some methods in
> > > ProcessSession
> > > > > > that
> > > > > > > > are
> > > > > > > > > > > never
> > > > > > > > > > > > > used
> > > > > > > > > > > > > > by any processor in the codebase but exists in
> the
> > > > public
> > > > > > API
> > > > > > > > so
> > > > > > > > > > > can’t
> > > > > > > > > > > > be
> > > > > > > > > > > > > > removed till 2.0.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think his is also a great time to clean up the
> > > > “standard
> > > > > > > > nar.”
> > > > > > > > > At
> > > > > > > > > > > > this
> > > > > > > > > > > > > > point, it’s something like 70 MB. And many of the
> > > > > > components
> > > > > > > > > there
> > > > > > > > > > > are
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > really “standard” - things like connecting to
> FTP &
> > > > SFTP
> > > > > > > > > servers, XML
> > > > > > > > > > > > > > processing, Jolt transform, etc. could
> potentially
> > be
> > > > moved
> > > > > > > > into
> > > > > > > > > > > other
> > > > > > > > > > > > > > nars. The
> > > > nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
> > > > > > is
> > > > > > > > > 6.9 MB
> > > > > > > > > > > is
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > necessary for stateless or minifi java. Lots of
> > > things
> > > > > > > probably
> > > > > > > > > to
> > > > > > > > > > > > > > reconsider within the standard nar.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I definitely think this is a reasonable approach,
> > to
> > > > allow
> > > > > > > for
> > > > > > > > a
> > > > > > > > > 2.0
> > > > > > > > > > > > that
> > > > > > > > > > > > > > is not a huge feature release but allows the
> > project
> > > to
> > > > be
> > > > > > > > > simpler
> > > > > > > > > > > and
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > nimble.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > -Mark
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > > > > > > > > mikerthomsen@gmail.com
> > > > > > > > > > > > > <mailto:
> > > > > > > > > > > > > > mikerthomsen@gmail.com>> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Russell,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > AFAICT from looking at Elastic's repos, the low
> > level
> > > > REST
> > > > > > > > > client is
> > > > > > > > > > > > > > still fine.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Our Elasticsearch support is spread over two NARs
> > at
> > > > > > present.
> > > > > > > > One
> > > > > > > > > > > uses
> > > > > > > > > > > > > > OkHttp and the other uses that low level Elastic
> > REST
> > > > > > client.
> > > > > > > > > > > > > > Therefore, I think we're fine on licensing for
> the
> > > > moment.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Mike
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > > > > > > > > > > russ@windofkeltia.com
> > > > > > > > > > > > > > <ma...@windofkeltia.com>> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Bringing up Elastic also reminds me that the
> > Elastic
> > > > > > > framework
> > > > > > > > > has
> > > > > > > > > > > just
> > > > > > > > > > > > > > recently transitioned out of Open Source, so to
> > > > acknowledge
> > > > > > > > that,
> > > > > > > > > > > maybe
> > > > > > > > > > > > > > some effort toward OpenSearch--I say this not
> > > > understanding
> > > > > > > > > exactly
> > > > > > > > > > > how
> > > > > > > > > > > > > > this sort of thing is considered in a
> large-scale,
> > > > > > > world-class
> > > > > > > > > > > software
> > > > > > > > > > > > > > project like Apache NiFi. (I'm not a contributor,
> > > just
> > > > a
> > > > > > > > grateful
> > > > > > > > > > > > > > consumer.)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Russ
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > > > > > > > > > > > > Along with the itemized list for ancient
> components
> > > we
> > > > > > should
> > > > > > > > > look at
> > > > > > > > > > > > > > updating versions of drivers, SDKs, etc. for
> > external
> > > > > > systems
> > > > > > > > > such as
> > > > > > > > > > > > > > Elasticsearch, Cassandra, etc. There may be
> > breaking
> > > > > > changes
> > > > > > > > but
> > > > > > > > > 2.0
> > > > > > > > > > > > > > is probably the right time to get things up to
> date
> > > to
> > > > make
> > > > > > > > them
> > > > > > > > > more
> > > > > > > > > > > > > > useful to more people.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > > > > > > > > thenatog@gmail.com
> > > > > > > > > > > > > <mailto:
> > > > > > > > > > > > > > thenatog@gmail.com>> wrote:
> > > > > > > > > > > > > > I'm a +1 for removing pretty much all of this
> > stuff.
> > > > There
> > > > > > > are
> > > > > > > > > > > security
> > > > > > > > > > > > > > implications to keeping old dependencies around,
> so
> > > the
> > > > > > more
> > > > > > > > old
> > > > > > > > > code
> > > > > > > > > > > > we
> > > > > > > > > > > > > > can remove the better. I agree that eventually we
> > > need
> > > > to
> > > > > > > move
> > > > > > > > to
> > > > > > > > > > > > > > supporting only Java 11+, and as our next release
> > > will
> > > > > > > probably
> > > > > > > > > be
> > > > > > > > > > > > about
> > > > > > > > > > > > > 4
> > > > > > > > > > > > > > - 6 months from now that doesn't seem too soon.
> We
> > > > could
> > > > > > > > > potentially
> > > > > > > > > > > > > break
> > > > > > > > > > > > > > this in two and remove the deprecated processors
> > and
> > > > leave
> > > > > > > 1.x
> > > > > > > > on
> > > > > > > > > > > Java
> > > > > > > > > > > > 8,
> > > > > > > > > > > > > > and finally start on 2.x which would support only
> > > Java
> > > > 11.
> > > > > > > I'm
> > > > > > > > > unsure
> > > > > > > > > > > > of
> > > > > > > > > > > > > > what implications changing the date and time
> > handling
> > > > would
> > > > > > > > have
> > > > > > > > > -
> > > > > > > > > > > for
> > > > > > > > > > > > > > running systems that use long term historical
> logs,
> > > > > > > unexpected
> > > > > > > > > > > impacts
> > > > > > > > > > > > to
> > > > > > > > > > > > > > time logging could be a problem.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > As Joe says I think feature work will have to be
> > > > dedicated
> > > > > > to
> > > > > > > > > 2.x and
> > > > > > > > > > > > we
> > > > > > > > > > > > > > could support 1.x for security fixes for some
> > period
> > > of
> > > > > > time.
> > > > > > > > 2.x
> > > > > > > > > > > seems
> > > > > > > > > > > > > > like a gargantuan task but it's probably time to
> > get
> > > > > > started.
> > > > > > > > Not
> > > > > > > > > > > sure
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > we handle all open PRs and the transition between
> > 1.x
> > > > and
> > > > > > > 2.x.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> > > > > > > joe.witt@gmail.com
> > > > > > > > > > > <mailto:
> > > > > > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jon
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You're right we have to be careful and you're
> right
> > > > there
> > > > > > are
> > > > > > > > > still
> > > > > > > > > > > > > > significant Java 8 users out there. But we also
> > have
> > > to
> > > > be
> > > > > > > > > careful
> > > > > > > > > > > > > > about security and sustainability of the
> codebase.
> > If
> > > > we
> > > > > > had
> > > > > > > > > talked
> > > > > > > > > > > > > > about this last year when that article came out
> I'd
> > > > have
> > > > > > > agreed
> > > > > > > > > it is
> > > > > > > > > > > > > > too early. Interestingly that link seems to get
> > > updated
> > > > > > and I
> > > > > > > > > tried
> > > > > > > > > > > > > > [1] and found more recent data (not sure how
> > recent).
> > > > > > Anyway
> > > > > > > it
> > > > > > > > > > > > > > suggests Java 8 is still the top dog but we see
> > good
> > > > growth
> > > > > > > on
> > > > > > > > > 11. In
> > > > > > > > > > > > > > my $dayjob this aligns to what I'm seeing too.
> > > > Customers
> > > > > > > didn't
> > > > > > > > > seem
> > > > > > > > > > > > > > to care about Java 11 until later half last year
> > and
> > > > now
> > > > > > > > > suddenly it
> > > > > > > > > > > > > > is all over the place.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think once we put out a NiFi 2.0 release we'd
> see
> > > > rapid
> > > > > > > > > decrease in
> > > > > > > > > > > > > > work on the 1.x line just being blunt. We did
> this
> > > many
> > > > > > years
> > > > > > > > ago
> > > > > > > > > > > > > > with 0.x to 1.x and we stood behind 0.x for a
> while
> > > > (maybe
> > > > > > a
> > > > > > > > > year or
> > > > > > > > > > > > > > so) but it was purely bug fix/security related
> > bits.
> > > We
> > > > > > would
> > > > > > > > > need to
> > > > > > > > > > > > > > do something similar. But feature work would
> almost
> > > > > > certainly
> > > > > > > > go
> > > > > > > > > to
> > > > > > > > > > > > > > the 2.x line. Maybe there are other workable
> models
> > > but
> > > > my
> > > > > > > > > instinct
> > > > > > > > > > > > > > suggests this is likely to follow a similar path.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ...anyway I agree it isn't that easy of a call to
> > > dump
> > > > Java
> > > > > > > 8.
> > > > > > > > We
> > > > > > > > > > > > > > need to make the call in both the interests of
> the
> > > user
> > > > > > base
> > > > > > > > and
> > > > > > > > > the
> > > > > > > > > > > > > > contributor base of the community.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1]
> > > https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > Joe
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
> > > > > > joe.witt@gmail.com
> > > > > > > > > <mailto:
> > > > > > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > > > > > > Russ
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yeah the flow registry is a key part of it. But
> > also
> > > > now
> > > > > > you
> > > > > > > > can
> > > > > > > > > > > > > > download the flow definition in JSON (upload i
> > think
> > > is
> > > > > > there
> > > > > > > > now
> > > > > > > > > > > > > > too). Templates offered a series of challenges
> such
> > > as
> > > > we
> > > > > > > store
> > > > > > > > > them
> > > > > > > > > > > > > > in the flow definition which has made flows
> massive
> > > in
> > > > an
> > > > > > > > > unintended
> > > > > > > > > > > > > > way which isn't fun for cluster behavior.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We have a couple cases where we headed down a
> > > > particular
> > > > > > > > concept
> > > > > > > > > and
> > > > > > > > > > > > > > came up with better approaches later. We need to
> > > > reconcile
> > > > > > > > these
> > > > > > > > > with
> > > > > > > > > > > > > > the benefit of hindsight, and while being careful
> > to
> > > be
> > > > not
> > > > > > > > > overly
> > > > > > > > > > > > > > disruptive to existing users, to reduce the
> > > > > > > > codebase/maintenance
> > > > > > > > > > > > > > burden and allow continued evolution of the
> > project.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > > > > > > > > > > russ@windofkeltia.com
> > > > > > > > > > > > > > <ma...@windofkeltia.com>>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > Joe,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I apologize for the off-topic intrusion, but what
> > > > replaces
> > > > > > > > > templates?
> > > > > > > > > > > > > > The Registry? Templates rocked and we have used
> > them
> > > > since
> > > > > > > > 0.5.x.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Russ
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > > > > > > > > > > David,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think this is a highly reasonable approach and
> > > such a
> > > > > > focus
> > > > > > > > > will
> > > > > > > > > > > > > > greatly help make a 2.0 release far more
> > approachable
> > > > to
> > > > > > > knock
> > > > > > > > > out.
> > > > > > > > > > > > > > Not only that but tech debt reduction would help
> > make
> > > > work
> > > > > > > > > towards
> > > > > > > > > > > > > > major features we'd think about in a 'major
> > release'
> > > > sense
> > > > > > > more
> > > > > > > > > > > > > > approachable.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We should remove all deprecated things (as well
> as
> > > > verify
> > > > > > we
> > > > > > > > > have the
> > > > > > > > > > > > > > right list). We should remove/consider removal of
> > > > > > deprecated
> > > > > > > > > > > > > > concepts
> > > > > > > > > > > > > > like templates. We should consider whether we can
> > > > resolve
> > > > > > the
> > > > > > > > > > > > > > various
> > > > > > > > > > > > > > ways we've handled what are now parameters down
> to
> > > one
> > > > > > clean
> > > > > > > > > > > > > > approach.
> > > > > > > > > > > > > > We should remove options in the nifi.properties
> > which
> > > > turn
> > > > > > > out
> > > > > > > > to
> > > > > > > > > > > > > > never be used quite right (if there are). There
> is
> > > > quite a
> > > > > > > bit
> > > > > > > > we
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > do purely in the name of tech debt reduction.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Lots to consider here but I think this is the
> right
> > > > > > > discussion.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Than ks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> > > > > > > bbende@gmail.com
> > > > > > > > > > > <mailto:
> > > > > > > > > > > > > > bbende@gmail.com>>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > I'm a +1 for this... Not sure if this falls under
> > > > "Removing
> > > > > > > > > > > > > > Deprecated
> > > > > > > > > > > > > > Components", but I think we should also look at
> > > > anything
> > > > > > that
> > > > > > > > has
> > > > > > > > > > > > > > been
> > > > > > > > > > > > > > marked as deprecated throughout the code base as
> a
> > > > > > candidate
> > > > > > > > for
> > > > > > > > > > > > > > removal. There are quite a few classes, methods,
> > > > > > properties,
> > > > > > > > etc
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > have been waiting for a chance to be removed.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > > > > > > > > > > > <exceptionfactory@apache.org<mailto:
> > > > > > > > exceptionfactory@apache.org
> > > > > > > > > >>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > Team,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > With all of the excellent work that many have
> > > > contributed
> > > > > > to
> > > > > > > > NiFi
> > > > > > > > > > > > > > over the
> > > > > > > > > > > > > > years, the code base has also accumulated some
> > amount
> > > > of
> > > > > > > > > technical
> > > > > > > > > > > > > > debt. A
> > > > > > > > > > > > > > handful of components have been marked as
> > deprecated,
> > > > and
> > > > > > > some
> > > > > > > > > > > > > > components
> > > > > > > > > > > > > > remain in the code base to support integration
> with
> > > old
> > > > > > > > versions
> > > > > > > > > > > > > > of various
> > > > > > > > > > > > > > products. Following the principles of semantic
> > > > versioning,
> > > > > > > > > > > > > > introducing a
> > > > > > > > > > > > > > major release would provide the opportunity to
> > remove
> > > > these
> > > > > > > > > > > > > > deprecated and
> > > > > > > > > > > > > > unsupported components.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Rather than focusing the next major release on
> new
> > > > > > features,
> > > > > > > > what
> > > > > > > > > > > > > > do you
> > > > > > > > > > > > > > think about focusing on technical debt removal?
> > This
> > > > > > approach
> > > > > > > > > > > > > > would not
> > > > > > > > > > > > > > make for the most interesting release, but it
> > > provides
> > > > the
> > > > > > > > > > > > > > opportunity to
> > > > > > > > > > > > > > clean up elements that involve breaking changes.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Focusing on technical debt, at least three
> primary
> > > > goals
> > > > > > come
> > > > > > > > to
> > > > > > > > > > > > > > mind for
> > > > > > > > > > > > > > the next major release:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. Removal of deprecated and unmaintained
> > components
> > > > > > > > > > > > > > 2. Require Java 11 as the minimum supported
> version
> > > > > > > > > > > > > > 3. Transition internal date and time handling to
> > JSR
> > > > 310
> > > > > > > > > java.time
> > > > > > > > > > > > > > components
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Removing Deprecated Components*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Removing support for older and deprecated
> > components
> > > > > > > provides a
> > > > > > > > > > > > > > great
> > > > > > > > > > > > > > opportunity to improve the overall security
> posture
> > > > when it
> > > > > > > > comes
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > maintaining dependencies. The OWASP dependency
> > plugin
> > > > > > report
> > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > generates 50 MB of HTML for questionable
> > > dependencies,
> > > > many
> > > > > > > of
> > > > > > > > > > > > > > which are
> > > > > > > > > > > > > > related to old versions of various libraries.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > As a starting point, here are a handful of
> > components
> > > > and
> > > > > > > > > > > > > > extension modules
> > > > > > > > > > > > > > that could be targeted for removal in a major
> > > version:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > - PostHTTP and GetHTTP
> > > > > > > > > > > > > > - ListenLumberjack and the entire
> > > > nifi-lumberjack-bundle
> > > > > > > > > > > > > > - ListenBeats and the entire nifi-beats-bundle
> > > > > > > > > > > > > > - Elasticsearch 5 components
> > > > > > > > > > > > > > - Hive 1 and 2 components
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Requiring Java 11*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Java 8 is now over seven years old, and NiFi has
> > > > supported
> > > > > > > > > general
> > > > > > > > > > > > > > compatibility with Java 11 for several years.
> NiFi
> > > > 1.14.0
> > > > > > > > > > > > > > incorporated
> > > > > > > > > > > > > > internal improvements specifically related to TLS
> > > 1.3,
> > > > > > which
> > > > > > > > > > > > > > allowed
> > > > > > > > > > > > > > closing out the long-running Java 11
> compatibility
> > > epic
> > > > > > > > > NIFI-5174.
> > > > > > > > > > > > > > Making
> > > > > > > > > > > > > > Java 11 the minimum required version provides the
> > > > > > opportunity
> > > > > > > > to
> > > > > > > > > > > > > > address
> > > > > > > > > > > > > > any lingering edge cases and put NiFi in a better
> > > > position
> > > > > > to
> > > > > > > > > > > > > > support
> > > > > > > > > > > > > > current Java versions.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *JSR 310 for Date and Time Handling*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Without making the scope too broad, transitioning
> > > > internal
> > > > > > > date
> > > > > > > > > > > > > > and time
> > > > > > > > > > > > > > handling to use DateTimeFormatter instead of
> > > > > > SimpleDateFormat
> > > > > > > > > > > > > > would provide
> > > > > > > > > > > > > > a number of advantages. The Java Time components
> > > > provide
> > > > > > much
> > > > > > > > > > > > > > better
> > > > > > > > > > > > > > clarity when it comes to handling localized date
> > and
> > > > time
> > > > > > > > > > > > > > representations,
> > > > > > > > > > > > > > and also avoid the inherent confusion of
> > > java.sql.Date
> > > > > > > > extending
> > > > > > > > > > > > > > java.util.Date. Many internal components,
> > > specifically
> > > > > > > > > > > > > > Record-oriented
> > > > > > > > > > > > > > processors and services, rely on date parsing,
> > > leading
> > > > to
> > > > > > > > > > > > > > confusion and
> > > > > > > > > > > > > > various workarounds. The pattern formats of
> > > > > > SimpleDateFormat
> > > > > > > > and
> > > > > > > > > > > > > > DateTimeFormatter are very similar, but there
> are a
> > > few
> > > > > > > subtle
> > > > > > > > > > > > > > differences.
> > > > > > > > > > > > > > Making this transition would provide a much
> better
> > > > > > foundation
> > > > > > > > > > > > > > going forward.
> > > > > > > > > > > > > > *Conclusion*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for giving this proposal some
> consideration.
> > > > Many of
> > > > > > > you
> > > > > > > > > > > > > > have been
> > > > > > > > > > > > > > developing NiFi for years and I look forward to
> > your
> > > > > > > feedback.
> > > > > > > > I
> > > > > > > > > > > > > > would be
> > > > > > > > > > > > > > glad to put together a more formalized
> > recommendation
> > > > on
> > > > > > > > > > > > > > Confluence and
> > > > > > > > > > > > > > write up Jira epics if this general approach
> sounds
> > > > > > agreeable
> > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > community.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > David Handermann
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > >
> >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Mark Bean <ma...@gmail.com>.
I created a JIRA ticket to investigate and improve the performance of
InvokeHTTP. It includes a flow definition for benchmarking the performance
of both PostHTTP and InvokeHTTP.
https://issues.apache.org/jira/browse/NIFI-8968
Thanks,
Mark
On Fri, Jul 30, 2021 at 6:41 PM Adam Taft <ad...@adamtaft.com> wrote:
> I'm not seeing the side thread that was going to discuss deprecation of
> PostHTTP. Has that thread started and I just don't see it?
>
> One (significant?) concern with PostHTTP is the smooth integration of
> NiFi-to-NiFi communication that is very transparently enabled with the
> ListenHTTP and PostHTTP processors. There's some special logic in there for
> handling flowfiles that InvokeHTTP doesn't really (nor should really) have.
>
> I know of several (large) NiFi installations that rely on the PostHTTP /
> ListenHTTP combination. It has enabled NiFi to NiFi communication for folks
> reluctant or unable to enable site-to-site type configuration.
>
> Honestly, I don't know why we'd want to "deprecate" any processor, as
> opposed to just moving it to a new location. If these processors can be
> ported and maintained to whatever the 2.0 API looks like, there's possibly
> little harm keeping them around.
>
> And by the way, what happened to the "marketplace" concept? Is this being
> considered for 2.0 as well? Because relocating the deprecated processors
> to an external nar might be the best solution. Losing PostHTTP entirely I
> think would be a mistake, but I'd conceptually support relocating it.
>
> Thanks,
>
> /Adam
>
> On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <jo...@gmail.com> wrote:
>
> > Looks like we just need to knock out 5 JIRAs :) [1]
> >
> > I felt like we had a label folks were using at one point but quickly
> > looking revealed nothing exciting. After this confluence page
> > stabilizes a bit we can probably knock out some JIRAs and such.
> >
> > [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
> >
> > On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <ot...@gmail.com>
> > wrote:
> > >
> > > I find myself wishing I had a list of all the jiras / issues that have
> > > been put off for a 2.0 release because they required some change or
> > another
> > > :(
> > >
> > > From: Joe Witt <jo...@gmail.com> <jo...@gmail.com>
> > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > > Date: July 27, 2021 at 12:30:35
> > > To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > >
> > > A few thoughts:
> > >
> > > 1. I would love to see deprecation notices show up in the UI in
> > > various ways to help motivate users to move off things to more
> > > supportable things. That is not a prerequisite for anything happening
> > > however. Just a good feature/nice thing to do for users when someone
> > > is able to tackle it.
> > >
> > > 2. The decision to deprecate something and to further remove it need
> > > not mean there is a superior solution available. If that thing itself
> > > isn't getting the love/attention it needs to be
> > > maintained/supported/healthy going forward that alone is enough to
> > > remove it. That might well be the case with PostHTTP [1] and for
> > > comparison you can see how much effort has gone into InvokeHTTP [2].
> > >
> > > 3. When discussing a 2.0 release each thing we add as a 'must do' the
> > > further away from reality such a release will become. We'll have to
> > > get very specific about 'musts' vs 'wants'.
> > >
> > > [1]
> > >
> >
> https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
> > > [2]
> > >
> >
> https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
> > >
> > > On Tue, Jul 27, 2021 at 8:47 AM David Handermann
> > > <ex...@apache.org> wrote:
> > > >
> > > > Thanks Mark, providing a template or comparison statistics with Java
> > > > versions and component configuration details would be very helpful.
> If
> > it
> > > > is possible to run tests using a public API or deployable service,
> that
> > > > would also help confirm potential differences.
> > > >
> > > > Showing a deprecation notice in the UI could be helpful, perhaps as a
> > > > configurable option. NIFI-8650 describes a general Flow Analysis
> > > > capability, and it seems like that might be one possible way to
> surface
> > > > deprecation warnings. For something more specific to component
> > > deprecation,
> > > > it seems important to find a balance between making it obvious and
> > making
> > > > it something that ends up getting ignored.
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <ma...@gmail.com>
> > wrote:
> > > >
> > > > > I'll start a new thread for PostHTTP when I get a template and/or
> > > detailed
> > > > > stats.
> > > > >
> > > > > I know the deprecation is noted in the documentation. That's a
> > > necessary
> > > > > and minimum level of notification. I was suggesting it be more
> > obvious
> > > in
> > > > > the UI. I think it would be beneficial to somehow be aware of the
> > > > > deprecation status simply by looking at the flow (perhaps even on
> the
> > > > > summary pages too), and without having to open the documentation
> for
> > > every
> > > > > processor to confirm whether or not a component is marked as
> > > deprecated.
> > > > >
> > > > > Thanks,
> > > > > Mark
> > > > >
> > > > >
> > > > > On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
> > > > > exceptionfactory@apache.org> wrote:
> > > > >
> > > > > > Mark,
> > > > > >
> > > > > > Thanks for the feedback. It may be better to start a separate
> > thread
> > > on
> > > > > > PostHTTP, but can you provide an example flow demonstrating the
> > > > > performance
> > > > > > differences between PostHTTP and InvokeHTTP?
> > > > > >
> > > > > > PostHTTP uses the Apache HttpComponents library, whereas
> InvokeHTTP
> > > uses
> > > > > > OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for
> > > OkHttp,
> > > > > > so it would be important to test with the most recent version. It
> > is
> > > also
> > > > > > worth noting that test classes for PostHTTP are now integration
> > tests
> > > as
> > > > > > opposed to unit tests, which means they are not executed as part
> of
> > > the
> > > > > > automated builds.
> > > > > >
> > > > > > The NiFi documentation includes the contents of the
> > DeprecationNotice
> > > for
> > > > > > PostHTTP:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > >
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
> > > > > >
> > > > > > Regards,
> > > > > > David Handermann
> > > > > >
> > > > > > On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <mark.o.bean@gmail.com
> >
> > > wrote:
> > > > > >
> > > > > > > I'm strongly in favor of reducing tech debt, and moving
> > > deliberately
> > > > > to a
> > > > > > > 2.0 release. I have a concern with just one processor that is
> > > currently
> > > > > > > marked as deprecated: PostHTTP. (I have not evaluated
> > specifically
> > > any
> > > > > > > other deprecated components; I cannot say if there are or are
> not
> > > > > similar
> > > > > > > issues elsewhere.) I understand the rationale for deprecating
> > this
> > > > > > > processor in that it eliminates a processor whose functionality
> > is
> > > > > > > available elsewhere, specifically in the more flexible
> > InvokeHTTP.
> > > > > > However,
> > > > > > > in my experience and testing, PostHTTP performs transfers with
> > far
> > > > > > greater
> > > > > > > throughput than InvokeHTTP. I would not be in favor of removing
> > > > > PostHTTP
> > > > > > > unless/until InvokeHTTP is refactored to increase its
> throughput
> > > > > > > capability.
> > > > > > >
> > > > > > > Has anyone else continued to use PostHTTP over InvokeHTTP for
> > > similar
> > > > > > > reasons? Or, is there a performance-related configuration for
> > > > > InvokeHTTP
> > > > > > I
> > > > > > > may have missed?
> > > > > > >
> > > > > > > Also, in order to help facilitate a smooth transition to 2.0
> > from a
> > > > > user
> > > > > > > perspective, would it be advisable to add some sort of visual
> > > indicator
> > > > > > in
> > > > > > > the UI for components that are currently annotated as
> > @Deprecated?
> > > This
> > > > > > > might help users proactively modify their flow prior to a
> release
> > > that
> > > > > > > would otherwise break it.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <bb...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > With the merging of MiNiFi and registry into the NiFi repo,
> > we've
> > > > > > > > actually gone more towards a mono-repo where everything is
> > > released
> > > > > > > > together. Eventually it would still be nice to have a smaller
> > > base
> > > > > > > > distribution containing just the framework and standard NARs,
> > but
> > > it
> > > > > > > > is hard to tackle that until we provide a way for users to
> > easily
> > > > > > > > obtain the NARs in some other way.
> > > > > > > >
> > > > > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <
> > > edward.armes@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Given the major version number shift and the spliting up of
> > > > > > processors
> > > > > > > > into
> > > > > > > > > multiple NAR's. I'd like to suggest that we start
> > individually
> > > > > > > versioning
> > > > > > > > > individual NARs/ bundles.
> > > > > > > > >
> > > > > > > > > I can see this bringing a large number of benifits
> including
> > > making
> > > > > > > Nifi
> > > > > > > > > more deployable with things RPM, but also potentially
> > allowing
> > > > > those
> > > > > > > that
> > > > > > > > > have to deploy managed Nifi instances easier to upgrade.
> > > > > > > > >
> > > > > > > > > Edward
> > > > > > > > >
> > > > > > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, <
> > ottobackwards@gmail.com>
> > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > The issue with updating the aws sdk is if it breaks any
> one
> > > of
> > > > > the
> > > > > > > > > > processors.
> > > > > > > > > > the Web Gateway API invoke processor for example is not
> > using
> > > a
> > > > > > high
> > > > > > > > level
> > > > > > > > > > purpose build client and may break.
> > > > > > > > > >
> > > > > > > > > > If we change the aws version, we need to coordinate in
> > such a
> > > way
> > > > > > > that
> > > > > > > > they
> > > > > > > > > > all
> > > > > > > > > > can come along reasonably.
> > > > > > > > > > IE: what happens if 1 or 2 break but the rest or OK?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > From: David Handermann <ex...@apache.org>
> > > > > > > > > > <ex...@apache.org>
> > > > > > > > > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > > > > dev@nifi.apache.org>
> > > > > > > > > > Date: July 26, 2021 at 09:33:42
> > > > > > > > > > To: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > > dev@nifi.apache.org
> > > > > > >
> > > > > > > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > > > > > > > >
> > > > > > > > > > Chris,
> > > > > > > > > >
> > > > > > > > > > Thanks for the reply and recommendations. It seems like
> > some
> > > of
> > > > > the
> > > > > > > > work to
> > > > > > > > > > reorganize the module structure could be done outside of
> a
> > > major
> > > > > > > > release,
> > > > > > > > > > but it would be great to target any breaking changes for
> > 2.0.
> > > > > > > Perhaps a
> > > > > > > > > > separate feature proposal on module restructuring, with
> the
> > > goal
> > > > > of
> > > > > > > > > > supporting optimized builds, would be a helpful way to
> move
> > > that
> > > > > > part
> > > > > > > > of
> > > > > > > > > > the discussion forward.
> > > > > > > > > >
> > > > > > > > > > Regarding updating AWS SDK to version 2, it seems like
> that
> > > might
> > > > > > be
> > > > > > > > > > possible now. I haven't taken a close look at the
> > referencing
> > > > > > > > components,
> > > > > > > > > > so I'm not sure about the level of effort involved. Minor
> > > NiFi
> > > > > > > version
> > > > > > > > > > updates have incorporated major new versions of
> > dependencies.
> > > For
> > > > > > > > example,
> > > > > > > > > > NiFi 1.14 included an upgrade from Spring Framework 4 to
> 5.
> > > On
> > > > > the
> > > > > > > one
> > > > > > > > > > hand, including the AWS SDK update as part of a major
> > release
> > > > > seems
> > > > > > > > > > helpful, but unless there are changes that break existing
> > > > > component
> > > > > > > > > > properties, upgrading the AWS SDK could be worked
> > > independently.
> > > > > > > > Others may
> > > > > > > > > > have more insight into particular usage of that library.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > David Handermann
> > > > > > > > > >
> > > > > > > > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > > > > > > > > > <ch...@naimuri.com.invalid> wrote:
> > > > > > > > > >
> > > > > > > > > > > Might be worth considering refactoring the build as
> part
> > of
> > > > > this
> > > > > > > work
> > > > > > > > > > too,
> > > > > > > > > > > e.g. only building the bits of the repo affected by a
> > > commit,
> > > > > > etc.
> > > > > > > -
> > > > > > > > > > > discussed briefly in previous threads but don't think
> any
> > > > > changes
> > > > > > > > made
> > > > > > > > > > yet.
> > > > > > > > > > > If NARs/components are likely to be split up and
> > refactored
> > > > > then
> > > > > > > such
> > > > > > > > > > work
> > > > > > > > > > > around the build probably makes sense to consider.
> > > > > > > > > > >
> > > > > > > > > > > I've a couple of PRs open that include updates to
> > > Elasticsearch
> > > > > > > > versions
> > > > > > > > > > > already, although I stopped at 7.10.2 (after which
> > Elastic
> > > > > > changed
> > > > > > > > > > licence)
> > > > > > > > > > > in case there were licence concerns. But more work can
> be
> > > done
> > > > > to
> > > > > > > > tidy up
> > > > > > > > > > > the processors, absolutely.
> > > > > > > > > > >
> > > > > > > > > > > AWS libraries to v2 would seem a sensible move and a
> > > refactor
> > > > > of
> > > > > > > > those
> > > > > > > > > > > processors as well.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris Sampson
> > > > > > > > > > >
> > > > > > > > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > > > > > > > > > exceptionfactory@apache.org>
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks for pointing out the standard NAR bundles,
> Mark.
> > > There
> > > > > > > are a
> > > > > > > > > > > number
> > > > > > > > > > > > of components in the standard NAR bundles with
> > particular
> > > > > > > > dependencies
> > > > > > > > > > > that
> > > > > > > > > > > > would make more sense in separate NARs. Reorganizing
> > the
> > > > > > standard
> > > > > > > > NAR
> > > > > > > > > > to
> > > > > > > > > > > > components with limited dependencies and wide
> > > applicability
> > > > > > would
> > > > > > > > > > > > definitely help with future maintenance.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > David Handermann
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> > > > > > > markap14@hotmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > There’s also some code that exists in order to
> > maintain
> > > > > > > backward
> > > > > > > > > > > > > compatibility in the repositories. I would very
> much
> > > like
> > > > > the
> > > > > > > > > > > > repositories
> > > > > > > > > > > > > to contain no unnecessary code. And swap file
> format
> > > > > supports
> > > > > > > > really
> > > > > > > > > > > old
> > > > > > > > > > > > > formats. And the old impls of the repositories
> > > themselves,
> > > > > > like
> > > > > > > > > > > > > PersistentProvRepo instead of WriteAheadProv Repo,
> > etc.
> > > > > Lots
> > > > > > of
> > > > > > > > stuff
> > > > > > > > > > > > there
> > > > > > > > > > > > > that can be removed. And some methods in
> > ProcessSession
> > > > > that
> > > > > > > are
> > > > > > > > > > never
> > > > > > > > > > > > used
> > > > > > > > > > > > > by any processor in the codebase but exists in the
> > > public
> > > > > API
> > > > > > > so
> > > > > > > > > > can’t
> > > > > > > > > > > be
> > > > > > > > > > > > > removed till 2.0.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think his is also a great time to clean up the
> > > “standard
> > > > > > > nar.”
> > > > > > > > At
> > > > > > > > > > > this
> > > > > > > > > > > > > point, it’s something like 70 MB. And many of the
> > > > > components
> > > > > > > > there
> > > > > > > > > > are
> > > > > > > > > > > > not
> > > > > > > > > > > > > really “standard” - things like connecting to FTP &
> > > SFTP
> > > > > > > > servers, XML
> > > > > > > > > > > > > processing, Jolt transform, etc. could potentially
> be
> > > moved
> > > > > > > into
> > > > > > > > > > other
> > > > > > > > > > > > > nars. The
> > > nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
> > > > > is
> > > > > > > > 6.9 MB
> > > > > > > > > > is
> > > > > > > > > > > > not
> > > > > > > > > > > > > necessary for stateless or minifi java. Lots of
> > things
> > > > > > probably
> > > > > > > > to
> > > > > > > > > > > > > reconsider within the standard nar.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I definitely think this is a reasonable approach,
> to
> > > allow
> > > > > > for
> > > > > > > a
> > > > > > > > 2.0
> > > > > > > > > > > that
> > > > > > > > > > > > > is not a huge feature release but allows the
> project
> > to
> > > be
> > > > > > > > simpler
> > > > > > > > > > and
> > > > > > > > > > > > more
> > > > > > > > > > > > > nimble.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > -Mark
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > > > > > > > mikerthomsen@gmail.com
> > > > > > > > > > > > <mailto:
> > > > > > > > > > > > > mikerthomsen@gmail.com>> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Russell,
> > > > > > > > > > > > >
> > > > > > > > > > > > > AFAICT from looking at Elastic's repos, the low
> level
> > > REST
> > > > > > > > client is
> > > > > > > > > > > > > still fine.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > > > > > > > > > > >
> > > > > > > > > > > > > Our Elasticsearch support is spread over two NARs
> at
> > > > > present.
> > > > > > > One
> > > > > > > > > > uses
> > > > > > > > > > > > > OkHttp and the other uses that low level Elastic
> REST
> > > > > client.
> > > > > > > > > > > > > Therefore, I think we're fine on licensing for the
> > > moment.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Mike
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > > > > > > > > > russ@windofkeltia.com
> > > > > > > > > > > > > <ma...@windofkeltia.com>> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Bringing up Elastic also reminds me that the
> Elastic
> > > > > > framework
> > > > > > > > has
> > > > > > > > > > just
> > > > > > > > > > > > > recently transitioned out of Open Source, so to
> > > acknowledge
> > > > > > > that,
> > > > > > > > > > maybe
> > > > > > > > > > > > > some effort toward OpenSearch--I say this not
> > > understanding
> > > > > > > > exactly
> > > > > > > > > > how
> > > > > > > > > > > > > this sort of thing is considered in a large-scale,
> > > > > > world-class
> > > > > > > > > > software
> > > > > > > > > > > > > project like Apache NiFi. (I'm not a contributor,
> > just
> > > a
> > > > > > > grateful
> > > > > > > > > > > > > consumer.)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Russ
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > > > > > > > > > > > Along with the itemized list for ancient components
> > we
> > > > > should
> > > > > > > > look at
> > > > > > > > > > > > > updating versions of drivers, SDKs, etc. for
> external
> > > > > systems
> > > > > > > > such as
> > > > > > > > > > > > > Elasticsearch, Cassandra, etc. There may be
> breaking
> > > > > changes
> > > > > > > but
> > > > > > > > 2.0
> > > > > > > > > > > > > is probably the right time to get things up to date
> > to
> > > make
> > > > > > > them
> > > > > > > > more
> > > > > > > > > > > > > useful to more people.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > > > > > > > thenatog@gmail.com
> > > > > > > > > > > > <mailto:
> > > > > > > > > > > > > thenatog@gmail.com>> wrote:
> > > > > > > > > > > > > I'm a +1 for removing pretty much all of this
> stuff.
> > > There
> > > > > > are
> > > > > > > > > > security
> > > > > > > > > > > > > implications to keeping old dependencies around, so
> > the
> > > > > more
> > > > > > > old
> > > > > > > > code
> > > > > > > > > > > we
> > > > > > > > > > > > > can remove the better. I agree that eventually we
> > need
> > > to
> > > > > > move
> > > > > > > to
> > > > > > > > > > > > > supporting only Java 11+, and as our next release
> > will
> > > > > > probably
> > > > > > > > be
> > > > > > > > > > > about
> > > > > > > > > > > > 4
> > > > > > > > > > > > > - 6 months from now that doesn't seem too soon. We
> > > could
> > > > > > > > potentially
> > > > > > > > > > > > break
> > > > > > > > > > > > > this in two and remove the deprecated processors
> and
> > > leave
> > > > > > 1.x
> > > > > > > on
> > > > > > > > > > Java
> > > > > > > > > > > 8,
> > > > > > > > > > > > > and finally start on 2.x which would support only
> > Java
> > > 11.
> > > > > > I'm
> > > > > > > > unsure
> > > > > > > > > > > of
> > > > > > > > > > > > > what implications changing the date and time
> handling
> > > would
> > > > > > > have
> > > > > > > > -
> > > > > > > > > > for
> > > > > > > > > > > > > running systems that use long term historical logs,
> > > > > > unexpected
> > > > > > > > > > impacts
> > > > > > > > > > > to
> > > > > > > > > > > > > time logging could be a problem.
> > > > > > > > > > > > >
> > > > > > > > > > > > > As Joe says I think feature work will have to be
> > > dedicated
> > > > > to
> > > > > > > > 2.x and
> > > > > > > > > > > we
> > > > > > > > > > > > > could support 1.x for security fixes for some
> period
> > of
> > > > > time.
> > > > > > > 2.x
> > > > > > > > > > seems
> > > > > > > > > > > > > like a gargantuan task but it's probably time to
> get
> > > > > started.
> > > > > > > Not
> > > > > > > > > > sure
> > > > > > > > > > > > how
> > > > > > > > > > > > > we handle all open PRs and the transition between
> 1.x
> > > and
> > > > > > 2.x.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> > > > > > joe.witt@gmail.com
> > > > > > > > > > <mailto:
> > > > > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jon
> > > > > > > > > > > > >
> > > > > > > > > > > > > You're right we have to be careful and you're right
> > > there
> > > > > are
> > > > > > > > still
> > > > > > > > > > > > > significant Java 8 users out there. But we also
> have
> > to
> > > be
> > > > > > > > careful
> > > > > > > > > > > > > about security and sustainability of the codebase.
> If
> > > we
> > > > > had
> > > > > > > > talked
> > > > > > > > > > > > > about this last year when that article came out I'd
> > > have
> > > > > > agreed
> > > > > > > > it is
> > > > > > > > > > > > > too early. Interestingly that link seems to get
> > updated
> > > > > and I
> > > > > > > > tried
> > > > > > > > > > > > > [1] and found more recent data (not sure how
> recent).
> > > > > Anyway
> > > > > > it
> > > > > > > > > > > > > suggests Java 8 is still the top dog but we see
> good
> > > growth
> > > > > > on
> > > > > > > > 11. In
> > > > > > > > > > > > > my $dayjob this aligns to what I'm seeing too.
> > > Customers
> > > > > > didn't
> > > > > > > > seem
> > > > > > > > > > > > > to care about Java 11 until later half last year
> and
> > > now
> > > > > > > > suddenly it
> > > > > > > > > > > > > is all over the place.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think once we put out a NiFi 2.0 release we'd see
> > > rapid
> > > > > > > > decrease in
> > > > > > > > > > > > > work on the 1.x line just being blunt. We did this
> > many
> > > > > years
> > > > > > > ago
> > > > > > > > > > > > > with 0.x to 1.x and we stood behind 0.x for a while
> > > (maybe
> > > > > a
> > > > > > > > year or
> > > > > > > > > > > > > so) but it was purely bug fix/security related
> bits.
> > We
> > > > > would
> > > > > > > > need to
> > > > > > > > > > > > > do something similar. But feature work would almost
> > > > > certainly
> > > > > > > go
> > > > > > > > to
> > > > > > > > > > > > > the 2.x line. Maybe there are other workable models
> > but
> > > my
> > > > > > > > instinct
> > > > > > > > > > > > > suggests this is likely to follow a similar path.
> > > > > > > > > > > > >
> > > > > > > > > > > > > ...anyway I agree it isn't that easy of a call to
> > dump
> > > Java
> > > > > > 8.
> > > > > > > We
> > > > > > > > > > > > > need to make the call in both the interests of the
> > user
> > > > > base
> > > > > > > and
> > > > > > > > the
> > > > > > > > > > > > > contributor base of the community.
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1]
> > https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > Joe
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
> > > > > joe.witt@gmail.com
> > > > > > > > <mailto:
> > > > > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > > > > > Russ
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yeah the flow registry is a key part of it. But
> also
> > > now
> > > > > you
> > > > > > > can
> > > > > > > > > > > > > download the flow definition in JSON (upload i
> think
> > is
> > > > > there
> > > > > > > now
> > > > > > > > > > > > > too). Templates offered a series of challenges such
> > as
> > > we
> > > > > > store
> > > > > > > > them
> > > > > > > > > > > > > in the flow definition which has made flows massive
> > in
> > > an
> > > > > > > > unintended
> > > > > > > > > > > > > way which isn't fun for cluster behavior.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We have a couple cases where we headed down a
> > > particular
> > > > > > > concept
> > > > > > > > and
> > > > > > > > > > > > > came up with better approaches later. We need to
> > > reconcile
> > > > > > > these
> > > > > > > > with
> > > > > > > > > > > > > the benefit of hindsight, and while being careful
> to
> > be
> > > not
> > > > > > > > overly
> > > > > > > > > > > > > disruptive to existing users, to reduce the
> > > > > > > codebase/maintenance
> > > > > > > > > > > > > burden and allow continued evolution of the
> project.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > > > > > > > > > russ@windofkeltia.com
> > > > > > > > > > > > > <ma...@windofkeltia.com>>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > Joe,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I apologize for the off-topic intrusion, but what
> > > replaces
> > > > > > > > templates?
> > > > > > > > > > > > > The Registry? Templates rocked and we have used
> them
> > > since
> > > > > > > 0.5.x.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Russ
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > > > > > > > > > David,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think this is a highly reasonable approach and
> > such a
> > > > > focus
> > > > > > > > will
> > > > > > > > > > > > > greatly help make a 2.0 release far more
> approachable
> > > to
> > > > > > knock
> > > > > > > > out.
> > > > > > > > > > > > > Not only that but tech debt reduction would help
> make
> > > work
> > > > > > > > towards
> > > > > > > > > > > > > major features we'd think about in a 'major
> release'
> > > sense
> > > > > > more
> > > > > > > > > > > > > approachable.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We should remove all deprecated things (as well as
> > > verify
> > > > > we
> > > > > > > > have the
> > > > > > > > > > > > > right list). We should remove/consider removal of
> > > > > deprecated
> > > > > > > > > > > > > concepts
> > > > > > > > > > > > > like templates. We should consider whether we can
> > > resolve
> > > > > the
> > > > > > > > > > > > > various
> > > > > > > > > > > > > ways we've handled what are now parameters down to
> > one
> > > > > clean
> > > > > > > > > > > > > approach.
> > > > > > > > > > > > > We should remove options in the nifi.properties
> which
> > > turn
> > > > > > out
> > > > > > > to
> > > > > > > > > > > > > never be used quite right (if there are). There is
> > > quite a
> > > > > > bit
> > > > > > > we
> > > > > > > > > > > > > can
> > > > > > > > > > > > > do purely in the name of tech debt reduction.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Lots to consider here but I think this is the right
> > > > > > discussion.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Than ks
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> > > > > > bbende@gmail.com
> > > > > > > > > > <mailto:
> > > > > > > > > > > > > bbende@gmail.com>>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > I'm a +1 for this... Not sure if this falls under
> > > "Removing
> > > > > > > > > > > > > Deprecated
> > > > > > > > > > > > > Components", but I think we should also look at
> > > anything
> > > > > that
> > > > > > > has
> > > > > > > > > > > > > been
> > > > > > > > > > > > > marked as deprecated throughout the code base as a
> > > > > candidate
> > > > > > > for
> > > > > > > > > > > > > removal. There are quite a few classes, methods,
> > > > > properties,
> > > > > > > etc
> > > > > > > > > > > > > that
> > > > > > > > > > > > > have been waiting for a chance to be removed.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > > > > > > > > > > <exceptionfactory@apache.org<mailto:
> > > > > > > exceptionfactory@apache.org
> > > > > > > > >>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > Team,
> > > > > > > > > > > > >
> > > > > > > > > > > > > With all of the excellent work that many have
> > > contributed
> > > > > to
> > > > > > > NiFi
> > > > > > > > > > > > > over the
> > > > > > > > > > > > > years, the code base has also accumulated some
> amount
> > > of
> > > > > > > > technical
> > > > > > > > > > > > > debt. A
> > > > > > > > > > > > > handful of components have been marked as
> deprecated,
> > > and
> > > > > > some
> > > > > > > > > > > > > components
> > > > > > > > > > > > > remain in the code base to support integration with
> > old
> > > > > > > versions
> > > > > > > > > > > > > of various
> > > > > > > > > > > > > products. Following the principles of semantic
> > > versioning,
> > > > > > > > > > > > > introducing a
> > > > > > > > > > > > > major release would provide the opportunity to
> remove
> > > these
> > > > > > > > > > > > > deprecated and
> > > > > > > > > > > > > unsupported components.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Rather than focusing the next major release on new
> > > > > features,
> > > > > > > what
> > > > > > > > > > > > > do you
> > > > > > > > > > > > > think about focusing on technical debt removal?
> This
> > > > > approach
> > > > > > > > > > > > > would not
> > > > > > > > > > > > > make for the most interesting release, but it
> > provides
> > > the
> > > > > > > > > > > > > opportunity to
> > > > > > > > > > > > > clean up elements that involve breaking changes.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Focusing on technical debt, at least three primary
> > > goals
> > > > > come
> > > > > > > to
> > > > > > > > > > > > > mind for
> > > > > > > > > > > > > the next major release:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Removal of deprecated and unmaintained
> components
> > > > > > > > > > > > > 2. Require Java 11 as the minimum supported version
> > > > > > > > > > > > > 3. Transition internal date and time handling to
> JSR
> > > 310
> > > > > > > > java.time
> > > > > > > > > > > > > components
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Removing Deprecated Components*
> > > > > > > > > > > > >
> > > > > > > > > > > > > Removing support for older and deprecated
> components
> > > > > > provides a
> > > > > > > > > > > > > great
> > > > > > > > > > > > > opportunity to improve the overall security posture
> > > when it
> > > > > > > comes
> > > > > > > > > > > > > to
> > > > > > > > > > > > > maintaining dependencies. The OWASP dependency
> plugin
> > > > > report
> > > > > > > > > > > > > currently
> > > > > > > > > > > > > generates 50 MB of HTML for questionable
> > dependencies,
> > > many
> > > > > > of
> > > > > > > > > > > > > which are
> > > > > > > > > > > > > related to old versions of various libraries.
> > > > > > > > > > > > >
> > > > > > > > > > > > > As a starting point, here are a handful of
> components
> > > and
> > > > > > > > > > > > > extension modules
> > > > > > > > > > > > > that could be targeted for removal in a major
> > version:
> > > > > > > > > > > > >
> > > > > > > > > > > > > - PostHTTP and GetHTTP
> > > > > > > > > > > > > - ListenLumberjack and the entire
> > > nifi-lumberjack-bundle
> > > > > > > > > > > > > - ListenBeats and the entire nifi-beats-bundle
> > > > > > > > > > > > > - Elasticsearch 5 components
> > > > > > > > > > > > > - Hive 1 and 2 components
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Requiring Java 11*
> > > > > > > > > > > > >
> > > > > > > > > > > > > Java 8 is now over seven years old, and NiFi has
> > > supported
> > > > > > > > general
> > > > > > > > > > > > > compatibility with Java 11 for several years. NiFi
> > > 1.14.0
> > > > > > > > > > > > > incorporated
> > > > > > > > > > > > > internal improvements specifically related to TLS
> > 1.3,
> > > > > which
> > > > > > > > > > > > > allowed
> > > > > > > > > > > > > closing out the long-running Java 11 compatibility
> > epic
> > > > > > > > NIFI-5174.
> > > > > > > > > > > > > Making
> > > > > > > > > > > > > Java 11 the minimum required version provides the
> > > > > opportunity
> > > > > > > to
> > > > > > > > > > > > > address
> > > > > > > > > > > > > any lingering edge cases and put NiFi in a better
> > > position
> > > > > to
> > > > > > > > > > > > > support
> > > > > > > > > > > > > current Java versions.
> > > > > > > > > > > > >
> > > > > > > > > > > > > *JSR 310 for Date and Time Handling*
> > > > > > > > > > > > >
> > > > > > > > > > > > > Without making the scope too broad, transitioning
> > > internal
> > > > > > date
> > > > > > > > > > > > > and time
> > > > > > > > > > > > > handling to use DateTimeFormatter instead of
> > > > > SimpleDateFormat
> > > > > > > > > > > > > would provide
> > > > > > > > > > > > > a number of advantages. The Java Time components
> > > provide
> > > > > much
> > > > > > > > > > > > > better
> > > > > > > > > > > > > clarity when it comes to handling localized date
> and
> > > time
> > > > > > > > > > > > > representations,
> > > > > > > > > > > > > and also avoid the inherent confusion of
> > java.sql.Date
> > > > > > > extending
> > > > > > > > > > > > > java.util.Date. Many internal components,
> > specifically
> > > > > > > > > > > > > Record-oriented
> > > > > > > > > > > > > processors and services, rely on date parsing,
> > leading
> > > to
> > > > > > > > > > > > > confusion and
> > > > > > > > > > > > > various workarounds. The pattern formats of
> > > > > SimpleDateFormat
> > > > > > > and
> > > > > > > > > > > > > DateTimeFormatter are very similar, but there are a
> > few
> > > > > > subtle
> > > > > > > > > > > > > differences.
> > > > > > > > > > > > > Making this transition would provide a much better
> > > > > foundation
> > > > > > > > > > > > > going forward.
> > > > > > > > > > > > > *Conclusion*
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for giving this proposal some consideration.
> > > Many of
> > > > > > you
> > > > > > > > > > > > > have been
> > > > > > > > > > > > > developing NiFi for years and I look forward to
> your
> > > > > > feedback.
> > > > > > > I
> > > > > > > > > > > > > would be
> > > > > > > > > > > > > glad to put together a more formalized
> recommendation
> > > on
> > > > > > > > > > > > > Confluence and
> > > > > > > > > > > > > write up Jira epics if this general approach sounds
> > > > > agreeable
> > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > community.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > David Handermann
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Adam Taft <ad...@adamtaft.com>.
Thanks for the reply Joe.
If the "marketplace" concept can be scoped to something like "not in core
NiFi but can be downloaded here", then it could fit in the definition of
achievable in 2.0. The concept of the marketplace being this magical
dynamic processor lookup service always felt a little too vaporware to me.
One thing that would help the NiFi build size in general is to really focus
on what needs to be in "core" and what can just be external nars that the
dataflow/systems manager can add to the 'extensions' directory. We've
struggled with having really large builds, so maybe this is an opportunity
to address that in 2.0.
Radical brainstorm warning:
Imagine a completely "thin" NiFi distribution. e.g. with no processors or
other components at all. Then a configuration manager could source
additional nar functionality from a different distribution source. We could
still build processor bundles, but they would just be completely separate
from the framework. Separate git repository, separate versioning, separate
download, etc.
This would be the baby steps towards the marketplace concept. Let the
ecosystem of processors and other components thrive outside of the main
framework distribution. In this way, things like PostHTTP could live on
easily without burdening the framework. And groups of similar processor
functionality can exist in separately managed nars.
Thanks for entertaining the thought at least.
/Adam
On Fri, Jul 30, 2021 at 4:50 PM Joe Witt <jo...@gmail.com> wrote:
> Adam
>
> In cases of ‘what happened’ it just hasnt happened. The ideas are still
> good.
>
> I do agree with the ‘why kill anything’ mentality. We can simply park
> these somewhere. That said there is an ongoing maint cost we do have to
> rationalize. Builds are longer, deps to maintain, etc..
>
> Also we need to prob pumps the breaks a bit on thoughs of removing or
> changing many things. If we do we may well get to 2.0 but nobody could
> adopt it. We have honestly a pretty massive deployment base and we cannot
> make it a pain for users. We got away with things at 0.x to 1.0 that we
> cannot get away with on 2.0
>
> Thanks
>
> On Fri, Jul 30, 2021 at 3:41 PM Adam Taft <ad...@adamtaft.com> wrote:
>
> > I'm not seeing the side thread that was going to discuss deprecation of
> > PostHTTP. Has that thread started and I just don't see it?
> >
> > One (significant?) concern with PostHTTP is the smooth integration of
> > NiFi-to-NiFi communication that is very transparently enabled with the
> > ListenHTTP and PostHTTP processors. There's some special logic in there
> for
> > handling flowfiles that InvokeHTTP doesn't really (nor should really)
> have.
> >
> > I know of several (large) NiFi installations that rely on the PostHTTP /
> > ListenHTTP combination. It has enabled NiFi to NiFi communication for
> folks
> > reluctant or unable to enable site-to-site type configuration.
> >
> > Honestly, I don't know why we'd want to "deprecate" any processor, as
> > opposed to just moving it to a new location. If these processors can be
> > ported and maintained to whatever the 2.0 API looks like, there's
> possibly
> > little harm keeping them around.
> >
> > And by the way, what happened to the "marketplace" concept? Is this being
> > considered for 2.0 as well? Because relocating the deprecated processors
> > to an external nar might be the best solution. Losing PostHTTP entirely I
> > think would be a mistake, but I'd conceptually support relocating it.
> >
> > Thanks,
> >
> > /Adam
> >
> > On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <jo...@gmail.com> wrote:
> >
> > > Looks like we just need to knock out 5 JIRAs :) [1]
> > >
> > > I felt like we had a label folks were using at one point but quickly
> > > looking revealed nothing exciting. After this confluence page
> > > stabilizes a bit we can probably knock out some JIRAs and such.
> > >
> > > [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
> > >
> > > On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <ot...@gmail.com>
> > > wrote:
> > > >
> > > > I find myself wishing I had a list of all the jiras / issues that
> have
> > > > been put off for a 2.0 release because they required some change or
> > > another
> > > > :(
> > > >
> > > > From: Joe Witt <jo...@gmail.com> <jo...@gmail.com>
> > > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> dev@nifi.apache.org>
> > > > Date: July 27, 2021 at 12:30:35
> > > > To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > >
> > > > A few thoughts:
> > > >
> > > > 1. I would love to see deprecation notices show up in the UI in
> > > > various ways to help motivate users to move off things to more
> > > > supportable things. That is not a prerequisite for anything happening
> > > > however. Just a good feature/nice thing to do for users when someone
> > > > is able to tackle it.
> > > >
> > > > 2. The decision to deprecate something and to further remove it need
> > > > not mean there is a superior solution available. If that thing itself
> > > > isn't getting the love/attention it needs to be
> > > > maintained/supported/healthy going forward that alone is enough to
> > > > remove it. That might well be the case with PostHTTP [1] and for
> > > > comparison you can see how much effort has gone into InvokeHTTP [2].
> > > >
> > > > 3. When discussing a 2.0 release each thing we add as a 'must do' the
> > > > further away from reality such a release will become. We'll have to
> > > > get very specific about 'musts' vs 'wants'.
> > > >
> > > > [1]
> > > >
> > >
> >
> https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
> > > > [2]
> > > >
> > >
> >
> https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
> > > >
> > > > On Tue, Jul 27, 2021 at 8:47 AM David Handermann
> > > > <ex...@apache.org> wrote:
> > > > >
> > > > > Thanks Mark, providing a template or comparison statistics with
> Java
> > > > > versions and component configuration details would be very helpful.
> > If
> > > it
> > > > > is possible to run tests using a public API or deployable service,
> > that
> > > > > would also help confirm potential differences.
> > > > >
> > > > > Showing a deprecation notice in the UI could be helpful, perhaps
> as a
> > > > > configurable option. NIFI-8650 describes a general Flow Analysis
> > > > > capability, and it seems like that might be one possible way to
> > surface
> > > > > deprecation warnings. For something more specific to component
> > > > deprecation,
> > > > > it seems important to find a balance between making it obvious and
> > > making
> > > > > it something that ends up getting ignored.
> > > > >
> > > > > Regards,
> > > > > David Handermann
> > > > >
> > > > > On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <ma...@gmail.com>
> > > wrote:
> > > > >
> > > > > > I'll start a new thread for PostHTTP when I get a template and/or
> > > > detailed
> > > > > > stats.
> > > > > >
> > > > > > I know the deprecation is noted in the documentation. That's a
> > > > necessary
> > > > > > and minimum level of notification. I was suggesting it be more
> > > obvious
> > > > in
> > > > > > the UI. I think it would be beneficial to somehow be aware of the
> > > > > > deprecation status simply by looking at the flow (perhaps even on
> > the
> > > > > > summary pages too), and without having to open the documentation
> > for
> > > > every
> > > > > > processor to confirm whether or not a component is marked as
> > > > deprecated.
> > > > > >
> > > > > > Thanks,
> > > > > > Mark
> > > > > >
> > > > > >
> > > > > > On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
> > > > > > exceptionfactory@apache.org> wrote:
> > > > > >
> > > > > > > Mark,
> > > > > > >
> > > > > > > Thanks for the feedback. It may be better to start a separate
> > > thread
> > > > on
> > > > > > > PostHTTP, but can you provide an example flow demonstrating the
> > > > > > performance
> > > > > > > differences between PostHTTP and InvokeHTTP?
> > > > > > >
> > > > > > > PostHTTP uses the Apache HttpComponents library, whereas
> > InvokeHTTP
> > > > uses
> > > > > > > OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates
> for
> > > > OkHttp,
> > > > > > > so it would be important to test with the most recent version.
> It
> > > is
> > > > also
> > > > > > > worth noting that test classes for PostHTTP are now integration
> > > tests
> > > > as
> > > > > > > opposed to unit tests, which means they are not executed as
> part
> > of
> > > > the
> > > > > > > automated builds.
> > > > > > >
> > > > > > > The NiFi documentation includes the contents of the
> > > DeprecationNotice
> > > > for
> > > > > > > PostHTTP:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
> > > > > > >
> > > > > > > Regards,
> > > > > > > David Handermann
> > > > > > >
> > > > > > > On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <
> mark.o.bean@gmail.com
> > >
> > > > wrote:
> > > > > > >
> > > > > > > > I'm strongly in favor of reducing tech debt, and moving
> > > > deliberately
> > > > > > to a
> > > > > > > > 2.0 release. I have a concern with just one processor that is
> > > > currently
> > > > > > > > marked as deprecated: PostHTTP. (I have not evaluated
> > > specifically
> > > > any
> > > > > > > > other deprecated components; I cannot say if there are or are
> > not
> > > > > > similar
> > > > > > > > issues elsewhere.) I understand the rationale for deprecating
> > > this
> > > > > > > > processor in that it eliminates a processor whose
> functionality
> > > is
> > > > > > > > available elsewhere, specifically in the more flexible
> > > InvokeHTTP.
> > > > > > > However,
> > > > > > > > in my experience and testing, PostHTTP performs transfers
> with
> > > far
> > > > > > > greater
> > > > > > > > throughput than InvokeHTTP. I would not be in favor of
> removing
> > > > > > PostHTTP
> > > > > > > > unless/until InvokeHTTP is refactored to increase its
> > throughput
> > > > > > > > capability.
> > > > > > > >
> > > > > > > > Has anyone else continued to use PostHTTP over InvokeHTTP for
> > > > similar
> > > > > > > > reasons? Or, is there a performance-related configuration for
> > > > > > InvokeHTTP
> > > > > > > I
> > > > > > > > may have missed?
> > > > > > > >
> > > > > > > > Also, in order to help facilitate a smooth transition to 2.0
> > > from a
> > > > > > user
> > > > > > > > perspective, would it be advisable to add some sort of visual
> > > > indicator
> > > > > > > in
> > > > > > > > the UI for components that are currently annotated as
> > > @Deprecated?
> > > > This
> > > > > > > > might help users proactively modify their flow prior to a
> > release
> > > > that
> > > > > > > > would otherwise break it.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <
> bbende@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > With the merging of MiNiFi and registry into the NiFi repo,
> > > we've
> > > > > > > > > actually gone more towards a mono-repo where everything is
> > > > released
> > > > > > > > > together. Eventually it would still be nice to have a
> smaller
> > > > base
> > > > > > > > > distribution containing just the framework and standard
> NARs,
> > > but
> > > > it
> > > > > > > > > is hard to tackle that until we provide a way for users to
> > > easily
> > > > > > > > > obtain the NARs in some other way.
> > > > > > > > >
> > > > > > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <
> > > > edward.armes@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Given the major version number shift and the spliting up
> of
> > > > > > > processors
> > > > > > > > > into
> > > > > > > > > > multiple NAR's. I'd like to suggest that we start
> > > individually
> > > > > > > > versioning
> > > > > > > > > > individual NARs/ bundles.
> > > > > > > > > >
> > > > > > > > > > I can see this bringing a large number of benifits
> > including
> > > > making
> > > > > > > > Nifi
> > > > > > > > > > more deployable with things RPM, but also potentially
> > > allowing
> > > > > > those
> > > > > > > > that
> > > > > > > > > > have to deploy managed Nifi instances easier to upgrade.
> > > > > > > > > >
> > > > > > > > > > Edward
> > > > > > > > > >
> > > > > > > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, <
> > > ottobackwards@gmail.com>
> > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > The issue with updating the aws sdk is if it breaks any
> > one
> > > > of
> > > > > > the
> > > > > > > > > > > processors.
> > > > > > > > > > > the Web Gateway API invoke processor for example is not
> > > using
> > > > a
> > > > > > > high
> > > > > > > > > level
> > > > > > > > > > > purpose build client and may break.
> > > > > > > > > > >
> > > > > > > > > > > If we change the aws version, we need to coordinate in
> > > such a
> > > > way
> > > > > > > > that
> > > > > > > > > they
> > > > > > > > > > > all
> > > > > > > > > > > can come along reasonably.
> > > > > > > > > > > IE: what happens if 1 or 2 break but the rest or OK?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > From: David Handermann <ex...@apache.org>
> > > > > > > > > > > <ex...@apache.org>
> > > > > > > > > > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > > > > > dev@nifi.apache.org>
> > > > > > > > > > > Date: July 26, 2021 at 09:33:42
> > > > > > > > > > > To: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > > > dev@nifi.apache.org
> > > > > > > >
> > > > > > > > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > > > > > > > > >
> > > > > > > > > > > Chris,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the reply and recommendations. It seems like
> > > some
> > > > of
> > > > > > the
> > > > > > > > > work to
> > > > > > > > > > > reorganize the module structure could be done outside
> of
> > a
> > > > major
> > > > > > > > > release,
> > > > > > > > > > > but it would be great to target any breaking changes
> for
> > > 2.0.
> > > > > > > > Perhaps a
> > > > > > > > > > > separate feature proposal on module restructuring, with
> > the
> > > > goal
> > > > > > of
> > > > > > > > > > > supporting optimized builds, would be a helpful way to
> > move
> > > > that
> > > > > > > part
> > > > > > > > > of
> > > > > > > > > > > the discussion forward.
> > > > > > > > > > >
> > > > > > > > > > > Regarding updating AWS SDK to version 2, it seems like
> > that
> > > > might
> > > > > > > be
> > > > > > > > > > > possible now. I haven't taken a close look at the
> > > referencing
> > > > > > > > > components,
> > > > > > > > > > > so I'm not sure about the level of effort involved.
> Minor
> > > > NiFi
> > > > > > > > version
> > > > > > > > > > > updates have incorporated major new versions of
> > > dependencies.
> > > > For
> > > > > > > > > example,
> > > > > > > > > > > NiFi 1.14 included an upgrade from Spring Framework 4
> to
> > 5.
> > > > On
> > > > > > the
> > > > > > > > one
> > > > > > > > > > > hand, including the AWS SDK update as part of a major
> > > release
> > > > > > seems
> > > > > > > > > > > helpful, but unless there are changes that break
> existing
> > > > > > component
> > > > > > > > > > > properties, upgrading the AWS SDK could be worked
> > > > independently.
> > > > > > > > > Others may
> > > > > > > > > > > have more insight into particular usage of that
> library.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > David Handermann
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > > > > > > > > > > <ch...@naimuri.com.invalid> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Might be worth considering refactoring the build as
> > part
> > > of
> > > > > > this
> > > > > > > > work
> > > > > > > > > > > too,
> > > > > > > > > > > > e.g. only building the bits of the repo affected by a
> > > > commit,
> > > > > > > etc.
> > > > > > > > -
> > > > > > > > > > > > discussed briefly in previous threads but don't think
> > any
> > > > > > changes
> > > > > > > > > made
> > > > > > > > > > > yet.
> > > > > > > > > > > > If NARs/components are likely to be split up and
> > > refactored
> > > > > > then
> > > > > > > > such
> > > > > > > > > > > work
> > > > > > > > > > > > around the build probably makes sense to consider.
> > > > > > > > > > > >
> > > > > > > > > > > > I've a couple of PRs open that include updates to
> > > > Elasticsearch
> > > > > > > > > versions
> > > > > > > > > > > > already, although I stopped at 7.10.2 (after which
> > > Elastic
> > > > > > > changed
> > > > > > > > > > > licence)
> > > > > > > > > > > > in case there were licence concerns. But more work
> can
> > be
> > > > done
> > > > > > to
> > > > > > > > > tidy up
> > > > > > > > > > > > the processors, absolutely.
> > > > > > > > > > > >
> > > > > > > > > > > > AWS libraries to v2 would seem a sensible move and a
> > > > refactor
> > > > > > of
> > > > > > > > > those
> > > > > > > > > > > > processors as well.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > >
> > > > > > > > > > > > Chris Sampson
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > > > > > > > > > > exceptionfactory@apache.org>
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for pointing out the standard NAR bundles,
> > Mark.
> > > > There
> > > > > > > > are a
> > > > > > > > > > > > number
> > > > > > > > > > > > > of components in the standard NAR bundles with
> > > particular
> > > > > > > > > dependencies
> > > > > > > > > > > > that
> > > > > > > > > > > > > would make more sense in separate NARs.
> Reorganizing
> > > the
> > > > > > > standard
> > > > > > > > > NAR
> > > > > > > > > > > to
> > > > > > > > > > > > > components with limited dependencies and wide
> > > > applicability
> > > > > > > would
> > > > > > > > > > > > > definitely help with future maintenance.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > David Handermann
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> > > > > > > > markap14@hotmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > There’s also some code that exists in order to
> > > maintain
> > > > > > > > backward
> > > > > > > > > > > > > > compatibility in the repositories. I would very
> > much
> > > > like
> > > > > > the
> > > > > > > > > > > > > repositories
> > > > > > > > > > > > > > to contain no unnecessary code. And swap file
> > format
> > > > > > supports
> > > > > > > > > really
> > > > > > > > > > > > old
> > > > > > > > > > > > > > formats. And the old impls of the repositories
> > > > themselves,
> > > > > > > like
> > > > > > > > > > > > > > PersistentProvRepo instead of WriteAheadProv
> Repo,
> > > etc.
> > > > > > Lots
> > > > > > > of
> > > > > > > > > stuff
> > > > > > > > > > > > > there
> > > > > > > > > > > > > > that can be removed. And some methods in
> > > ProcessSession
> > > > > > that
> > > > > > > > are
> > > > > > > > > > > never
> > > > > > > > > > > > > used
> > > > > > > > > > > > > > by any processor in the codebase but exists in
> the
> > > > public
> > > > > > API
> > > > > > > > so
> > > > > > > > > > > can’t
> > > > > > > > > > > > be
> > > > > > > > > > > > > > removed till 2.0.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think his is also a great time to clean up the
> > > > “standard
> > > > > > > > nar.”
> > > > > > > > > At
> > > > > > > > > > > > this
> > > > > > > > > > > > > > point, it’s something like 70 MB. And many of the
> > > > > > components
> > > > > > > > > there
> > > > > > > > > > > are
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > really “standard” - things like connecting to
> FTP &
> > > > SFTP
> > > > > > > > > servers, XML
> > > > > > > > > > > > > > processing, Jolt transform, etc. could
> potentially
> > be
> > > > moved
> > > > > > > > into
> > > > > > > > > > > other
> > > > > > > > > > > > > > nars. The
> > > > nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
> > > > > > is
> > > > > > > > > 6.9 MB
> > > > > > > > > > > is
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > necessary for stateless or minifi java. Lots of
> > > things
> > > > > > > probably
> > > > > > > > > to
> > > > > > > > > > > > > > reconsider within the standard nar.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I definitely think this is a reasonable approach,
> > to
> > > > allow
> > > > > > > for
> > > > > > > > a
> > > > > > > > > 2.0
> > > > > > > > > > > > that
> > > > > > > > > > > > > > is not a huge feature release but allows the
> > project
> > > to
> > > > be
> > > > > > > > > simpler
> > > > > > > > > > > and
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > nimble.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > -Mark
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > > > > > > > > mikerthomsen@gmail.com
> > > > > > > > > > > > > <mailto:
> > > > > > > > > > > > > > mikerthomsen@gmail.com>> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Russell,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > AFAICT from looking at Elastic's repos, the low
> > level
> > > > REST
> > > > > > > > > client is
> > > > > > > > > > > > > > still fine.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Our Elasticsearch support is spread over two NARs
> > at
> > > > > > present.
> > > > > > > > One
> > > > > > > > > > > uses
> > > > > > > > > > > > > > OkHttp and the other uses that low level Elastic
> > REST
> > > > > > client.
> > > > > > > > > > > > > > Therefore, I think we're fine on licensing for
> the
> > > > moment.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Mike
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > > > > > > > > > > russ@windofkeltia.com
> > > > > > > > > > > > > > <ma...@windofkeltia.com>> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Bringing up Elastic also reminds me that the
> > Elastic
> > > > > > > framework
> > > > > > > > > has
> > > > > > > > > > > just
> > > > > > > > > > > > > > recently transitioned out of Open Source, so to
> > > > acknowledge
> > > > > > > > that,
> > > > > > > > > > > maybe
> > > > > > > > > > > > > > some effort toward OpenSearch--I say this not
> > > > understanding
> > > > > > > > > exactly
> > > > > > > > > > > how
> > > > > > > > > > > > > > this sort of thing is considered in a
> large-scale,
> > > > > > > world-class
> > > > > > > > > > > software
> > > > > > > > > > > > > > project like Apache NiFi. (I'm not a contributor,
> > > just
> > > > a
> > > > > > > > grateful
> > > > > > > > > > > > > > consumer.)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Russ
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > > > > > > > > > > > > Along with the itemized list for ancient
> components
> > > we
> > > > > > should
> > > > > > > > > look at
> > > > > > > > > > > > > > updating versions of drivers, SDKs, etc. for
> > external
> > > > > > systems
> > > > > > > > > such as
> > > > > > > > > > > > > > Elasticsearch, Cassandra, etc. There may be
> > breaking
> > > > > > changes
> > > > > > > > but
> > > > > > > > > 2.0
> > > > > > > > > > > > > > is probably the right time to get things up to
> date
> > > to
> > > > make
> > > > > > > > them
> > > > > > > > > more
> > > > > > > > > > > > > > useful to more people.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > > > > > > > > thenatog@gmail.com
> > > > > > > > > > > > > <mailto:
> > > > > > > > > > > > > > thenatog@gmail.com>> wrote:
> > > > > > > > > > > > > > I'm a +1 for removing pretty much all of this
> > stuff.
> > > > There
> > > > > > > are
> > > > > > > > > > > security
> > > > > > > > > > > > > > implications to keeping old dependencies around,
> so
> > > the
> > > > > > more
> > > > > > > > old
> > > > > > > > > code
> > > > > > > > > > > > we
> > > > > > > > > > > > > > can remove the better. I agree that eventually we
> > > need
> > > > to
> > > > > > > move
> > > > > > > > to
> > > > > > > > > > > > > > supporting only Java 11+, and as our next release
> > > will
> > > > > > > probably
> > > > > > > > > be
> > > > > > > > > > > > about
> > > > > > > > > > > > > 4
> > > > > > > > > > > > > > - 6 months from now that doesn't seem too soon.
> We
> > > > could
> > > > > > > > > potentially
> > > > > > > > > > > > > break
> > > > > > > > > > > > > > this in two and remove the deprecated processors
> > and
> > > > leave
> > > > > > > 1.x
> > > > > > > > on
> > > > > > > > > > > Java
> > > > > > > > > > > > 8,
> > > > > > > > > > > > > > and finally start on 2.x which would support only
> > > Java
> > > > 11.
> > > > > > > I'm
> > > > > > > > > unsure
> > > > > > > > > > > > of
> > > > > > > > > > > > > > what implications changing the date and time
> > handling
> > > > would
> > > > > > > > have
> > > > > > > > > -
> > > > > > > > > > > for
> > > > > > > > > > > > > > running systems that use long term historical
> logs,
> > > > > > > unexpected
> > > > > > > > > > > impacts
> > > > > > > > > > > > to
> > > > > > > > > > > > > > time logging could be a problem.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > As Joe says I think feature work will have to be
> > > > dedicated
> > > > > > to
> > > > > > > > > 2.x and
> > > > > > > > > > > > we
> > > > > > > > > > > > > > could support 1.x for security fixes for some
> > period
> > > of
> > > > > > time.
> > > > > > > > 2.x
> > > > > > > > > > > seems
> > > > > > > > > > > > > > like a gargantuan task but it's probably time to
> > get
> > > > > > started.
> > > > > > > > Not
> > > > > > > > > > > sure
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > we handle all open PRs and the transition between
> > 1.x
> > > > and
> > > > > > > 2.x.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> > > > > > > joe.witt@gmail.com
> > > > > > > > > > > <mailto:
> > > > > > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jon
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You're right we have to be careful and you're
> right
> > > > there
> > > > > > are
> > > > > > > > > still
> > > > > > > > > > > > > > significant Java 8 users out there. But we also
> > have
> > > to
> > > > be
> > > > > > > > > careful
> > > > > > > > > > > > > > about security and sustainability of the
> codebase.
> > If
> > > > we
> > > > > > had
> > > > > > > > > talked
> > > > > > > > > > > > > > about this last year when that article came out
> I'd
> > > > have
> > > > > > > agreed
> > > > > > > > > it is
> > > > > > > > > > > > > > too early. Interestingly that link seems to get
> > > updated
> > > > > > and I
> > > > > > > > > tried
> > > > > > > > > > > > > > [1] and found more recent data (not sure how
> > recent).
> > > > > > Anyway
> > > > > > > it
> > > > > > > > > > > > > > suggests Java 8 is still the top dog but we see
> > good
> > > > growth
> > > > > > > on
> > > > > > > > > 11. In
> > > > > > > > > > > > > > my $dayjob this aligns to what I'm seeing too.
> > > > Customers
> > > > > > > didn't
> > > > > > > > > seem
> > > > > > > > > > > > > > to care about Java 11 until later half last year
> > and
> > > > now
> > > > > > > > > suddenly it
> > > > > > > > > > > > > > is all over the place.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think once we put out a NiFi 2.0 release we'd
> see
> > > > rapid
> > > > > > > > > decrease in
> > > > > > > > > > > > > > work on the 1.x line just being blunt. We did
> this
> > > many
> > > > > > years
> > > > > > > > ago
> > > > > > > > > > > > > > with 0.x to 1.x and we stood behind 0.x for a
> while
> > > > (maybe
> > > > > > a
> > > > > > > > > year or
> > > > > > > > > > > > > > so) but it was purely bug fix/security related
> > bits.
> > > We
> > > > > > would
> > > > > > > > > need to
> > > > > > > > > > > > > > do something similar. But feature work would
> almost
> > > > > > certainly
> > > > > > > > go
> > > > > > > > > to
> > > > > > > > > > > > > > the 2.x line. Maybe there are other workable
> models
> > > but
> > > > my
> > > > > > > > > instinct
> > > > > > > > > > > > > > suggests this is likely to follow a similar path.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ...anyway I agree it isn't that easy of a call to
> > > dump
> > > > Java
> > > > > > > 8.
> > > > > > > > We
> > > > > > > > > > > > > > need to make the call in both the interests of
> the
> > > user
> > > > > > base
> > > > > > > > and
> > > > > > > > > the
> > > > > > > > > > > > > > contributor base of the community.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1]
> > > https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > Joe
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
> > > > > > joe.witt@gmail.com
> > > > > > > > > <mailto:
> > > > > > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > > > > > > Russ
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yeah the flow registry is a key part of it. But
> > also
> > > > now
> > > > > > you
> > > > > > > > can
> > > > > > > > > > > > > > download the flow definition in JSON (upload i
> > think
> > > is
> > > > > > there
> > > > > > > > now
> > > > > > > > > > > > > > too). Templates offered a series of challenges
> such
> > > as
> > > > we
> > > > > > > store
> > > > > > > > > them
> > > > > > > > > > > > > > in the flow definition which has made flows
> massive
> > > in
> > > > an
> > > > > > > > > unintended
> > > > > > > > > > > > > > way which isn't fun for cluster behavior.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We have a couple cases where we headed down a
> > > > particular
> > > > > > > > concept
> > > > > > > > > and
> > > > > > > > > > > > > > came up with better approaches later. We need to
> > > > reconcile
> > > > > > > > these
> > > > > > > > > with
> > > > > > > > > > > > > > the benefit of hindsight, and while being careful
> > to
> > > be
> > > > not
> > > > > > > > > overly
> > > > > > > > > > > > > > disruptive to existing users, to reduce the
> > > > > > > > codebase/maintenance
> > > > > > > > > > > > > > burden and allow continued evolution of the
> > project.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > > > > > > > > > > russ@windofkeltia.com
> > > > > > > > > > > > > > <ma...@windofkeltia.com>>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > Joe,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I apologize for the off-topic intrusion, but what
> > > > replaces
> > > > > > > > > templates?
> > > > > > > > > > > > > > The Registry? Templates rocked and we have used
> > them
> > > > since
> > > > > > > > 0.5.x.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Russ
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > > > > > > > > > > David,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think this is a highly reasonable approach and
> > > such a
> > > > > > focus
> > > > > > > > > will
> > > > > > > > > > > > > > greatly help make a 2.0 release far more
> > approachable
> > > > to
> > > > > > > knock
> > > > > > > > > out.
> > > > > > > > > > > > > > Not only that but tech debt reduction would help
> > make
> > > > work
> > > > > > > > > towards
> > > > > > > > > > > > > > major features we'd think about in a 'major
> > release'
> > > > sense
> > > > > > > more
> > > > > > > > > > > > > > approachable.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We should remove all deprecated things (as well
> as
> > > > verify
> > > > > > we
> > > > > > > > > have the
> > > > > > > > > > > > > > right list). We should remove/consider removal of
> > > > > > deprecated
> > > > > > > > > > > > > > concepts
> > > > > > > > > > > > > > like templates. We should consider whether we can
> > > > resolve
> > > > > > the
> > > > > > > > > > > > > > various
> > > > > > > > > > > > > > ways we've handled what are now parameters down
> to
> > > one
> > > > > > clean
> > > > > > > > > > > > > > approach.
> > > > > > > > > > > > > > We should remove options in the nifi.properties
> > which
> > > > turn
> > > > > > > out
> > > > > > > > to
> > > > > > > > > > > > > > never be used quite right (if there are). There
> is
> > > > quite a
> > > > > > > bit
> > > > > > > > we
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > do purely in the name of tech debt reduction.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Lots to consider here but I think this is the
> right
> > > > > > > discussion.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Than ks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> > > > > > > bbende@gmail.com
> > > > > > > > > > > <mailto:
> > > > > > > > > > > > > > bbende@gmail.com>>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > I'm a +1 for this... Not sure if this falls under
> > > > "Removing
> > > > > > > > > > > > > > Deprecated
> > > > > > > > > > > > > > Components", but I think we should also look at
> > > > anything
> > > > > > that
> > > > > > > > has
> > > > > > > > > > > > > > been
> > > > > > > > > > > > > > marked as deprecated throughout the code base as
> a
> > > > > > candidate
> > > > > > > > for
> > > > > > > > > > > > > > removal. There are quite a few classes, methods,
> > > > > > properties,
> > > > > > > > etc
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > have been waiting for a chance to be removed.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > > > > > > > > > > > <exceptionfactory@apache.org<mailto:
> > > > > > > > exceptionfactory@apache.org
> > > > > > > > > >>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > Team,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > With all of the excellent work that many have
> > > > contributed
> > > > > > to
> > > > > > > > NiFi
> > > > > > > > > > > > > > over the
> > > > > > > > > > > > > > years, the code base has also accumulated some
> > amount
> > > > of
> > > > > > > > > technical
> > > > > > > > > > > > > > debt. A
> > > > > > > > > > > > > > handful of components have been marked as
> > deprecated,
> > > > and
> > > > > > > some
> > > > > > > > > > > > > > components
> > > > > > > > > > > > > > remain in the code base to support integration
> with
> > > old
> > > > > > > > versions
> > > > > > > > > > > > > > of various
> > > > > > > > > > > > > > products. Following the principles of semantic
> > > > versioning,
> > > > > > > > > > > > > > introducing a
> > > > > > > > > > > > > > major release would provide the opportunity to
> > remove
> > > > these
> > > > > > > > > > > > > > deprecated and
> > > > > > > > > > > > > > unsupported components.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Rather than focusing the next major release on
> new
> > > > > > features,
> > > > > > > > what
> > > > > > > > > > > > > > do you
> > > > > > > > > > > > > > think about focusing on technical debt removal?
> > This
> > > > > > approach
> > > > > > > > > > > > > > would not
> > > > > > > > > > > > > > make for the most interesting release, but it
> > > provides
> > > > the
> > > > > > > > > > > > > > opportunity to
> > > > > > > > > > > > > > clean up elements that involve breaking changes.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Focusing on technical debt, at least three
> primary
> > > > goals
> > > > > > come
> > > > > > > > to
> > > > > > > > > > > > > > mind for
> > > > > > > > > > > > > > the next major release:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. Removal of deprecated and unmaintained
> > components
> > > > > > > > > > > > > > 2. Require Java 11 as the minimum supported
> version
> > > > > > > > > > > > > > 3. Transition internal date and time handling to
> > JSR
> > > > 310
> > > > > > > > > java.time
> > > > > > > > > > > > > > components
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Removing Deprecated Components*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Removing support for older and deprecated
> > components
> > > > > > > provides a
> > > > > > > > > > > > > > great
> > > > > > > > > > > > > > opportunity to improve the overall security
> posture
> > > > when it
> > > > > > > > comes
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > maintaining dependencies. The OWASP dependency
> > plugin
> > > > > > report
> > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > generates 50 MB of HTML for questionable
> > > dependencies,
> > > > many
> > > > > > > of
> > > > > > > > > > > > > > which are
> > > > > > > > > > > > > > related to old versions of various libraries.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > As a starting point, here are a handful of
> > components
> > > > and
> > > > > > > > > > > > > > extension modules
> > > > > > > > > > > > > > that could be targeted for removal in a major
> > > version:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > - PostHTTP and GetHTTP
> > > > > > > > > > > > > > - ListenLumberjack and the entire
> > > > nifi-lumberjack-bundle
> > > > > > > > > > > > > > - ListenBeats and the entire nifi-beats-bundle
> > > > > > > > > > > > > > - Elasticsearch 5 components
> > > > > > > > > > > > > > - Hive 1 and 2 components
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Requiring Java 11*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Java 8 is now over seven years old, and NiFi has
> > > > supported
> > > > > > > > > general
> > > > > > > > > > > > > > compatibility with Java 11 for several years.
> NiFi
> > > > 1.14.0
> > > > > > > > > > > > > > incorporated
> > > > > > > > > > > > > > internal improvements specifically related to TLS
> > > 1.3,
> > > > > > which
> > > > > > > > > > > > > > allowed
> > > > > > > > > > > > > > closing out the long-running Java 11
> compatibility
> > > epic
> > > > > > > > > NIFI-5174.
> > > > > > > > > > > > > > Making
> > > > > > > > > > > > > > Java 11 the minimum required version provides the
> > > > > > opportunity
> > > > > > > > to
> > > > > > > > > > > > > > address
> > > > > > > > > > > > > > any lingering edge cases and put NiFi in a better
> > > > position
> > > > > > to
> > > > > > > > > > > > > > support
> > > > > > > > > > > > > > current Java versions.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *JSR 310 for Date and Time Handling*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Without making the scope too broad, transitioning
> > > > internal
> > > > > > > date
> > > > > > > > > > > > > > and time
> > > > > > > > > > > > > > handling to use DateTimeFormatter instead of
> > > > > > SimpleDateFormat
> > > > > > > > > > > > > > would provide
> > > > > > > > > > > > > > a number of advantages. The Java Time components
> > > > provide
> > > > > > much
> > > > > > > > > > > > > > better
> > > > > > > > > > > > > > clarity when it comes to handling localized date
> > and
> > > > time
> > > > > > > > > > > > > > representations,
> > > > > > > > > > > > > > and also avoid the inherent confusion of
> > > java.sql.Date
> > > > > > > > extending
> > > > > > > > > > > > > > java.util.Date. Many internal components,
> > > specifically
> > > > > > > > > > > > > > Record-oriented
> > > > > > > > > > > > > > processors and services, rely on date parsing,
> > > leading
> > > > to
> > > > > > > > > > > > > > confusion and
> > > > > > > > > > > > > > various workarounds. The pattern formats of
> > > > > > SimpleDateFormat
> > > > > > > > and
> > > > > > > > > > > > > > DateTimeFormatter are very similar, but there
> are a
> > > few
> > > > > > > subtle
> > > > > > > > > > > > > > differences.
> > > > > > > > > > > > > > Making this transition would provide a much
> better
> > > > > > foundation
> > > > > > > > > > > > > > going forward.
> > > > > > > > > > > > > > *Conclusion*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for giving this proposal some
> consideration.
> > > > Many of
> > > > > > > you
> > > > > > > > > > > > > > have been
> > > > > > > > > > > > > > developing NiFi for years and I look forward to
> > your
> > > > > > > feedback.
> > > > > > > > I
> > > > > > > > > > > > > > would be
> > > > > > > > > > > > > > glad to put together a more formalized
> > recommendation
> > > > on
> > > > > > > > > > > > > > Confluence and
> > > > > > > > > > > > > > write up Jira epics if this general approach
> sounds
> > > > > > agreeable
> > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > community.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > David Handermann
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > >
> >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Joe Witt <jo...@gmail.com>.
Adam
In cases of ‘what happened’ it just hasnt happened. The ideas are still
good.
I do agree with the ‘why kill anything’ mentality. We can simply park
these somewhere. That said there is an ongoing maint cost we do have to
rationalize. Builds are longer, deps to maintain, etc..
Also we need to prob pumps the breaks a bit on thoughs of removing or
changing many things. If we do we may well get to 2.0 but nobody could
adopt it. We have honestly a pretty massive deployment base and we cannot
make it a pain for users. We got away with things at 0.x to 1.0 that we
cannot get away with on 2.0
Thanks
On Fri, Jul 30, 2021 at 3:41 PM Adam Taft <ad...@adamtaft.com> wrote:
> I'm not seeing the side thread that was going to discuss deprecation of
> PostHTTP. Has that thread started and I just don't see it?
>
> One (significant?) concern with PostHTTP is the smooth integration of
> NiFi-to-NiFi communication that is very transparently enabled with the
> ListenHTTP and PostHTTP processors. There's some special logic in there for
> handling flowfiles that InvokeHTTP doesn't really (nor should really) have.
>
> I know of several (large) NiFi installations that rely on the PostHTTP /
> ListenHTTP combination. It has enabled NiFi to NiFi communication for folks
> reluctant or unable to enable site-to-site type configuration.
>
> Honestly, I don't know why we'd want to "deprecate" any processor, as
> opposed to just moving it to a new location. If these processors can be
> ported and maintained to whatever the 2.0 API looks like, there's possibly
> little harm keeping them around.
>
> And by the way, what happened to the "marketplace" concept? Is this being
> considered for 2.0 as well? Because relocating the deprecated processors
> to an external nar might be the best solution. Losing PostHTTP entirely I
> think would be a mistake, but I'd conceptually support relocating it.
>
> Thanks,
>
> /Adam
>
> On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <jo...@gmail.com> wrote:
>
> > Looks like we just need to knock out 5 JIRAs :) [1]
> >
> > I felt like we had a label folks were using at one point but quickly
> > looking revealed nothing exciting. After this confluence page
> > stabilizes a bit we can probably knock out some JIRAs and such.
> >
> > [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
> >
> > On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <ot...@gmail.com>
> > wrote:
> > >
> > > I find myself wishing I had a list of all the jiras / issues that have
> > > been put off for a 2.0 release because they required some change or
> > another
> > > :(
> > >
> > > From: Joe Witt <jo...@gmail.com> <jo...@gmail.com>
> > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > > Date: July 27, 2021 at 12:30:35
> > > To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > >
> > > A few thoughts:
> > >
> > > 1. I would love to see deprecation notices show up in the UI in
> > > various ways to help motivate users to move off things to more
> > > supportable things. That is not a prerequisite for anything happening
> > > however. Just a good feature/nice thing to do for users when someone
> > > is able to tackle it.
> > >
> > > 2. The decision to deprecate something and to further remove it need
> > > not mean there is a superior solution available. If that thing itself
> > > isn't getting the love/attention it needs to be
> > > maintained/supported/healthy going forward that alone is enough to
> > > remove it. That might well be the case with PostHTTP [1] and for
> > > comparison you can see how much effort has gone into InvokeHTTP [2].
> > >
> > > 3. When discussing a 2.0 release each thing we add as a 'must do' the
> > > further away from reality such a release will become. We'll have to
> > > get very specific about 'musts' vs 'wants'.
> > >
> > > [1]
> > >
> >
> https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
> > > [2]
> > >
> >
> https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
> > >
> > > On Tue, Jul 27, 2021 at 8:47 AM David Handermann
> > > <ex...@apache.org> wrote:
> > > >
> > > > Thanks Mark, providing a template or comparison statistics with Java
> > > > versions and component configuration details would be very helpful.
> If
> > it
> > > > is possible to run tests using a public API or deployable service,
> that
> > > > would also help confirm potential differences.
> > > >
> > > > Showing a deprecation notice in the UI could be helpful, perhaps as a
> > > > configurable option. NIFI-8650 describes a general Flow Analysis
> > > > capability, and it seems like that might be one possible way to
> surface
> > > > deprecation warnings. For something more specific to component
> > > deprecation,
> > > > it seems important to find a balance between making it obvious and
> > making
> > > > it something that ends up getting ignored.
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <ma...@gmail.com>
> > wrote:
> > > >
> > > > > I'll start a new thread for PostHTTP when I get a template and/or
> > > detailed
> > > > > stats.
> > > > >
> > > > > I know the deprecation is noted in the documentation. That's a
> > > necessary
> > > > > and minimum level of notification. I was suggesting it be more
> > obvious
> > > in
> > > > > the UI. I think it would be beneficial to somehow be aware of the
> > > > > deprecation status simply by looking at the flow (perhaps even on
> the
> > > > > summary pages too), and without having to open the documentation
> for
> > > every
> > > > > processor to confirm whether or not a component is marked as
> > > deprecated.
> > > > >
> > > > > Thanks,
> > > > > Mark
> > > > >
> > > > >
> > > > > On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
> > > > > exceptionfactory@apache.org> wrote:
> > > > >
> > > > > > Mark,
> > > > > >
> > > > > > Thanks for the feedback. It may be better to start a separate
> > thread
> > > on
> > > > > > PostHTTP, but can you provide an example flow demonstrating the
> > > > > performance
> > > > > > differences between PostHTTP and InvokeHTTP?
> > > > > >
> > > > > > PostHTTP uses the Apache HttpComponents library, whereas
> InvokeHTTP
> > > uses
> > > > > > OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for
> > > OkHttp,
> > > > > > so it would be important to test with the most recent version. It
> > is
> > > also
> > > > > > worth noting that test classes for PostHTTP are now integration
> > tests
> > > as
> > > > > > opposed to unit tests, which means they are not executed as part
> of
> > > the
> > > > > > automated builds.
> > > > > >
> > > > > > The NiFi documentation includes the contents of the
> > DeprecationNotice
> > > for
> > > > > > PostHTTP:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > >
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
> > > > > >
> > > > > > Regards,
> > > > > > David Handermann
> > > > > >
> > > > > > On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <mark.o.bean@gmail.com
> >
> > > wrote:
> > > > > >
> > > > > > > I'm strongly in favor of reducing tech debt, and moving
> > > deliberately
> > > > > to a
> > > > > > > 2.0 release. I have a concern with just one processor that is
> > > currently
> > > > > > > marked as deprecated: PostHTTP. (I have not evaluated
> > specifically
> > > any
> > > > > > > other deprecated components; I cannot say if there are or are
> not
> > > > > similar
> > > > > > > issues elsewhere.) I understand the rationale for deprecating
> > this
> > > > > > > processor in that it eliminates a processor whose functionality
> > is
> > > > > > > available elsewhere, specifically in the more flexible
> > InvokeHTTP.
> > > > > > However,
> > > > > > > in my experience and testing, PostHTTP performs transfers with
> > far
> > > > > > greater
> > > > > > > throughput than InvokeHTTP. I would not be in favor of removing
> > > > > PostHTTP
> > > > > > > unless/until InvokeHTTP is refactored to increase its
> throughput
> > > > > > > capability.
> > > > > > >
> > > > > > > Has anyone else continued to use PostHTTP over InvokeHTTP for
> > > similar
> > > > > > > reasons? Or, is there a performance-related configuration for
> > > > > InvokeHTTP
> > > > > > I
> > > > > > > may have missed?
> > > > > > >
> > > > > > > Also, in order to help facilitate a smooth transition to 2.0
> > from a
> > > > > user
> > > > > > > perspective, would it be advisable to add some sort of visual
> > > indicator
> > > > > > in
> > > > > > > the UI for components that are currently annotated as
> > @Deprecated?
> > > This
> > > > > > > might help users proactively modify their flow prior to a
> release
> > > that
> > > > > > > would otherwise break it.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <bb...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > With the merging of MiNiFi and registry into the NiFi repo,
> > we've
> > > > > > > > actually gone more towards a mono-repo where everything is
> > > released
> > > > > > > > together. Eventually it would still be nice to have a smaller
> > > base
> > > > > > > > distribution containing just the framework and standard NARs,
> > but
> > > it
> > > > > > > > is hard to tackle that until we provide a way for users to
> > easily
> > > > > > > > obtain the NARs in some other way.
> > > > > > > >
> > > > > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <
> > > edward.armes@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Given the major version number shift and the spliting up of
> > > > > > processors
> > > > > > > > into
> > > > > > > > > multiple NAR's. I'd like to suggest that we start
> > individually
> > > > > > > versioning
> > > > > > > > > individual NARs/ bundles.
> > > > > > > > >
> > > > > > > > > I can see this bringing a large number of benifits
> including
> > > making
> > > > > > > Nifi
> > > > > > > > > more deployable with things RPM, but also potentially
> > allowing
> > > > > those
> > > > > > > that
> > > > > > > > > have to deploy managed Nifi instances easier to upgrade.
> > > > > > > > >
> > > > > > > > > Edward
> > > > > > > > >
> > > > > > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, <
> > ottobackwards@gmail.com>
> > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > The issue with updating the aws sdk is if it breaks any
> one
> > > of
> > > > > the
> > > > > > > > > > processors.
> > > > > > > > > > the Web Gateway API invoke processor for example is not
> > using
> > > a
> > > > > > high
> > > > > > > > level
> > > > > > > > > > purpose build client and may break.
> > > > > > > > > >
> > > > > > > > > > If we change the aws version, we need to coordinate in
> > such a
> > > way
> > > > > > > that
> > > > > > > > they
> > > > > > > > > > all
> > > > > > > > > > can come along reasonably.
> > > > > > > > > > IE: what happens if 1 or 2 break but the rest or OK?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > From: David Handermann <ex...@apache.org>
> > > > > > > > > > <ex...@apache.org>
> > > > > > > > > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > > > > dev@nifi.apache.org>
> > > > > > > > > > Date: July 26, 2021 at 09:33:42
> > > > > > > > > > To: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > > dev@nifi.apache.org
> > > > > > >
> > > > > > > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > > > > > > > >
> > > > > > > > > > Chris,
> > > > > > > > > >
> > > > > > > > > > Thanks for the reply and recommendations. It seems like
> > some
> > > of
> > > > > the
> > > > > > > > work to
> > > > > > > > > > reorganize the module structure could be done outside of
> a
> > > major
> > > > > > > > release,
> > > > > > > > > > but it would be great to target any breaking changes for
> > 2.0.
> > > > > > > Perhaps a
> > > > > > > > > > separate feature proposal on module restructuring, with
> the
> > > goal
> > > > > of
> > > > > > > > > > supporting optimized builds, would be a helpful way to
> move
> > > that
> > > > > > part
> > > > > > > > of
> > > > > > > > > > the discussion forward.
> > > > > > > > > >
> > > > > > > > > > Regarding updating AWS SDK to version 2, it seems like
> that
> > > might
> > > > > > be
> > > > > > > > > > possible now. I haven't taken a close look at the
> > referencing
> > > > > > > > components,
> > > > > > > > > > so I'm not sure about the level of effort involved. Minor
> > > NiFi
> > > > > > > version
> > > > > > > > > > updates have incorporated major new versions of
> > dependencies.
> > > For
> > > > > > > > example,
> > > > > > > > > > NiFi 1.14 included an upgrade from Spring Framework 4 to
> 5.
> > > On
> > > > > the
> > > > > > > one
> > > > > > > > > > hand, including the AWS SDK update as part of a major
> > release
> > > > > seems
> > > > > > > > > > helpful, but unless there are changes that break existing
> > > > > component
> > > > > > > > > > properties, upgrading the AWS SDK could be worked
> > > independently.
> > > > > > > > Others may
> > > > > > > > > > have more insight into particular usage of that library.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > David Handermann
> > > > > > > > > >
> > > > > > > > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > > > > > > > > > <ch...@naimuri.com.invalid> wrote:
> > > > > > > > > >
> > > > > > > > > > > Might be worth considering refactoring the build as
> part
> > of
> > > > > this
> > > > > > > work
> > > > > > > > > > too,
> > > > > > > > > > > e.g. only building the bits of the repo affected by a
> > > commit,
> > > > > > etc.
> > > > > > > -
> > > > > > > > > > > discussed briefly in previous threads but don't think
> any
> > > > > changes
> > > > > > > > made
> > > > > > > > > > yet.
> > > > > > > > > > > If NARs/components are likely to be split up and
> > refactored
> > > > > then
> > > > > > > such
> > > > > > > > > > work
> > > > > > > > > > > around the build probably makes sense to consider.
> > > > > > > > > > >
> > > > > > > > > > > I've a couple of PRs open that include updates to
> > > Elasticsearch
> > > > > > > > versions
> > > > > > > > > > > already, although I stopped at 7.10.2 (after which
> > Elastic
> > > > > > changed
> > > > > > > > > > licence)
> > > > > > > > > > > in case there were licence concerns. But more work can
> be
> > > done
> > > > > to
> > > > > > > > tidy up
> > > > > > > > > > > the processors, absolutely.
> > > > > > > > > > >
> > > > > > > > > > > AWS libraries to v2 would seem a sensible move and a
> > > refactor
> > > > > of
> > > > > > > > those
> > > > > > > > > > > processors as well.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris Sampson
> > > > > > > > > > >
> > > > > > > > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > > > > > > > > > exceptionfactory@apache.org>
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks for pointing out the standard NAR bundles,
> Mark.
> > > There
> > > > > > > are a
> > > > > > > > > > > number
> > > > > > > > > > > > of components in the standard NAR bundles with
> > particular
> > > > > > > > dependencies
> > > > > > > > > > > that
> > > > > > > > > > > > would make more sense in separate NARs. Reorganizing
> > the
> > > > > > standard
> > > > > > > > NAR
> > > > > > > > > > to
> > > > > > > > > > > > components with limited dependencies and wide
> > > applicability
> > > > > > would
> > > > > > > > > > > > definitely help with future maintenance.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > David Handermann
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> > > > > > > markap14@hotmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > There’s also some code that exists in order to
> > maintain
> > > > > > > backward
> > > > > > > > > > > > > compatibility in the repositories. I would very
> much
> > > like
> > > > > the
> > > > > > > > > > > > repositories
> > > > > > > > > > > > > to contain no unnecessary code. And swap file
> format
> > > > > supports
> > > > > > > > really
> > > > > > > > > > > old
> > > > > > > > > > > > > formats. And the old impls of the repositories
> > > themselves,
> > > > > > like
> > > > > > > > > > > > > PersistentProvRepo instead of WriteAheadProv Repo,
> > etc.
> > > > > Lots
> > > > > > of
> > > > > > > > stuff
> > > > > > > > > > > > there
> > > > > > > > > > > > > that can be removed. And some methods in
> > ProcessSession
> > > > > that
> > > > > > > are
> > > > > > > > > > never
> > > > > > > > > > > > used
> > > > > > > > > > > > > by any processor in the codebase but exists in the
> > > public
> > > > > API
> > > > > > > so
> > > > > > > > > > can’t
> > > > > > > > > > > be
> > > > > > > > > > > > > removed till 2.0.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think his is also a great time to clean up the
> > > “standard
> > > > > > > nar.”
> > > > > > > > At
> > > > > > > > > > > this
> > > > > > > > > > > > > point, it’s something like 70 MB. And many of the
> > > > > components
> > > > > > > > there
> > > > > > > > > > are
> > > > > > > > > > > > not
> > > > > > > > > > > > > really “standard” - things like connecting to FTP &
> > > SFTP
> > > > > > > > servers, XML
> > > > > > > > > > > > > processing, Jolt transform, etc. could potentially
> be
> > > moved
> > > > > > > into
> > > > > > > > > > other
> > > > > > > > > > > > > nars. The
> > > nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
> > > > > is
> > > > > > > > 6.9 MB
> > > > > > > > > > is
> > > > > > > > > > > > not
> > > > > > > > > > > > > necessary for stateless or minifi java. Lots of
> > things
> > > > > > probably
> > > > > > > > to
> > > > > > > > > > > > > reconsider within the standard nar.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I definitely think this is a reasonable approach,
> to
> > > allow
> > > > > > for
> > > > > > > a
> > > > > > > > 2.0
> > > > > > > > > > > that
> > > > > > > > > > > > > is not a huge feature release but allows the
> project
> > to
> > > be
> > > > > > > > simpler
> > > > > > > > > > and
> > > > > > > > > > > > more
> > > > > > > > > > > > > nimble.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > -Mark
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > > > > > > > mikerthomsen@gmail.com
> > > > > > > > > > > > <mailto:
> > > > > > > > > > > > > mikerthomsen@gmail.com>> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Russell,
> > > > > > > > > > > > >
> > > > > > > > > > > > > AFAICT from looking at Elastic's repos, the low
> level
> > > REST
> > > > > > > > client is
> > > > > > > > > > > > > still fine.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > > > > > > > > > > >
> > > > > > > > > > > > > Our Elasticsearch support is spread over two NARs
> at
> > > > > present.
> > > > > > > One
> > > > > > > > > > uses
> > > > > > > > > > > > > OkHttp and the other uses that low level Elastic
> REST
> > > > > client.
> > > > > > > > > > > > > Therefore, I think we're fine on licensing for the
> > > moment.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Mike
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > > > > > > > > > russ@windofkeltia.com
> > > > > > > > > > > > > <ma...@windofkeltia.com>> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Bringing up Elastic also reminds me that the
> Elastic
> > > > > > framework
> > > > > > > > has
> > > > > > > > > > just
> > > > > > > > > > > > > recently transitioned out of Open Source, so to
> > > acknowledge
> > > > > > > that,
> > > > > > > > > > maybe
> > > > > > > > > > > > > some effort toward OpenSearch--I say this not
> > > understanding
> > > > > > > > exactly
> > > > > > > > > > how
> > > > > > > > > > > > > this sort of thing is considered in a large-scale,
> > > > > > world-class
> > > > > > > > > > software
> > > > > > > > > > > > > project like Apache NiFi. (I'm not a contributor,
> > just
> > > a
> > > > > > > grateful
> > > > > > > > > > > > > consumer.)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Russ
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > > > > > > > > > > > Along with the itemized list for ancient components
> > we
> > > > > should
> > > > > > > > look at
> > > > > > > > > > > > > updating versions of drivers, SDKs, etc. for
> external
> > > > > systems
> > > > > > > > such as
> > > > > > > > > > > > > Elasticsearch, Cassandra, etc. There may be
> breaking
> > > > > changes
> > > > > > > but
> > > > > > > > 2.0
> > > > > > > > > > > > > is probably the right time to get things up to date
> > to
> > > make
> > > > > > > them
> > > > > > > > more
> > > > > > > > > > > > > useful to more people.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > > > > > > > thenatog@gmail.com
> > > > > > > > > > > > <mailto:
> > > > > > > > > > > > > thenatog@gmail.com>> wrote:
> > > > > > > > > > > > > I'm a +1 for removing pretty much all of this
> stuff.
> > > There
> > > > > > are
> > > > > > > > > > security
> > > > > > > > > > > > > implications to keeping old dependencies around, so
> > the
> > > > > more
> > > > > > > old
> > > > > > > > code
> > > > > > > > > > > we
> > > > > > > > > > > > > can remove the better. I agree that eventually we
> > need
> > > to
> > > > > > move
> > > > > > > to
> > > > > > > > > > > > > supporting only Java 11+, and as our next release
> > will
> > > > > > probably
> > > > > > > > be
> > > > > > > > > > > about
> > > > > > > > > > > > 4
> > > > > > > > > > > > > - 6 months from now that doesn't seem too soon. We
> > > could
> > > > > > > > potentially
> > > > > > > > > > > > break
> > > > > > > > > > > > > this in two and remove the deprecated processors
> and
> > > leave
> > > > > > 1.x
> > > > > > > on
> > > > > > > > > > Java
> > > > > > > > > > > 8,
> > > > > > > > > > > > > and finally start on 2.x which would support only
> > Java
> > > 11.
> > > > > > I'm
> > > > > > > > unsure
> > > > > > > > > > > of
> > > > > > > > > > > > > what implications changing the date and time
> handling
> > > would
> > > > > > > have
> > > > > > > > -
> > > > > > > > > > for
> > > > > > > > > > > > > running systems that use long term historical logs,
> > > > > > unexpected
> > > > > > > > > > impacts
> > > > > > > > > > > to
> > > > > > > > > > > > > time logging could be a problem.
> > > > > > > > > > > > >
> > > > > > > > > > > > > As Joe says I think feature work will have to be
> > > dedicated
> > > > > to
> > > > > > > > 2.x and
> > > > > > > > > > > we
> > > > > > > > > > > > > could support 1.x for security fixes for some
> period
> > of
> > > > > time.
> > > > > > > 2.x
> > > > > > > > > > seems
> > > > > > > > > > > > > like a gargantuan task but it's probably time to
> get
> > > > > started.
> > > > > > > Not
> > > > > > > > > > sure
> > > > > > > > > > > > how
> > > > > > > > > > > > > we handle all open PRs and the transition between
> 1.x
> > > and
> > > > > > 2.x.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> > > > > > joe.witt@gmail.com
> > > > > > > > > > <mailto:
> > > > > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jon
> > > > > > > > > > > > >
> > > > > > > > > > > > > You're right we have to be careful and you're right
> > > there
> > > > > are
> > > > > > > > still
> > > > > > > > > > > > > significant Java 8 users out there. But we also
> have
> > to
> > > be
> > > > > > > > careful
> > > > > > > > > > > > > about security and sustainability of the codebase.
> If
> > > we
> > > > > had
> > > > > > > > talked
> > > > > > > > > > > > > about this last year when that article came out I'd
> > > have
> > > > > > agreed
> > > > > > > > it is
> > > > > > > > > > > > > too early. Interestingly that link seems to get
> > updated
> > > > > and I
> > > > > > > > tried
> > > > > > > > > > > > > [1] and found more recent data (not sure how
> recent).
> > > > > Anyway
> > > > > > it
> > > > > > > > > > > > > suggests Java 8 is still the top dog but we see
> good
> > > growth
> > > > > > on
> > > > > > > > 11. In
> > > > > > > > > > > > > my $dayjob this aligns to what I'm seeing too.
> > > Customers
> > > > > > didn't
> > > > > > > > seem
> > > > > > > > > > > > > to care about Java 11 until later half last year
> and
> > > now
> > > > > > > > suddenly it
> > > > > > > > > > > > > is all over the place.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think once we put out a NiFi 2.0 release we'd see
> > > rapid
> > > > > > > > decrease in
> > > > > > > > > > > > > work on the 1.x line just being blunt. We did this
> > many
> > > > > years
> > > > > > > ago
> > > > > > > > > > > > > with 0.x to 1.x and we stood behind 0.x for a while
> > > (maybe
> > > > > a
> > > > > > > > year or
> > > > > > > > > > > > > so) but it was purely bug fix/security related
> bits.
> > We
> > > > > would
> > > > > > > > need to
> > > > > > > > > > > > > do something similar. But feature work would almost
> > > > > certainly
> > > > > > > go
> > > > > > > > to
> > > > > > > > > > > > > the 2.x line. Maybe there are other workable models
> > but
> > > my
> > > > > > > > instinct
> > > > > > > > > > > > > suggests this is likely to follow a similar path.
> > > > > > > > > > > > >
> > > > > > > > > > > > > ...anyway I agree it isn't that easy of a call to
> > dump
> > > Java
> > > > > > 8.
> > > > > > > We
> > > > > > > > > > > > > need to make the call in both the interests of the
> > user
> > > > > base
> > > > > > > and
> > > > > > > > the
> > > > > > > > > > > > > contributor base of the community.
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1]
> > https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > Joe
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
> > > > > joe.witt@gmail.com
> > > > > > > > <mailto:
> > > > > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > > > > > Russ
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yeah the flow registry is a key part of it. But
> also
> > > now
> > > > > you
> > > > > > > can
> > > > > > > > > > > > > download the flow definition in JSON (upload i
> think
> > is
> > > > > there
> > > > > > > now
> > > > > > > > > > > > > too). Templates offered a series of challenges such
> > as
> > > we
> > > > > > store
> > > > > > > > them
> > > > > > > > > > > > > in the flow definition which has made flows massive
> > in
> > > an
> > > > > > > > unintended
> > > > > > > > > > > > > way which isn't fun for cluster behavior.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We have a couple cases where we headed down a
> > > particular
> > > > > > > concept
> > > > > > > > and
> > > > > > > > > > > > > came up with better approaches later. We need to
> > > reconcile
> > > > > > > these
> > > > > > > > with
> > > > > > > > > > > > > the benefit of hindsight, and while being careful
> to
> > be
> > > not
> > > > > > > > overly
> > > > > > > > > > > > > disruptive to existing users, to reduce the
> > > > > > > codebase/maintenance
> > > > > > > > > > > > > burden and allow continued evolution of the
> project.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > > > > > > > > > russ@windofkeltia.com
> > > > > > > > > > > > > <ma...@windofkeltia.com>>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > Joe,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I apologize for the off-topic intrusion, but what
> > > replaces
> > > > > > > > templates?
> > > > > > > > > > > > > The Registry? Templates rocked and we have used
> them
> > > since
> > > > > > > 0.5.x.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Russ
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > > > > > > > > > David,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think this is a highly reasonable approach and
> > such a
> > > > > focus
> > > > > > > > will
> > > > > > > > > > > > > greatly help make a 2.0 release far more
> approachable
> > > to
> > > > > > knock
> > > > > > > > out.
> > > > > > > > > > > > > Not only that but tech debt reduction would help
> make
> > > work
> > > > > > > > towards
> > > > > > > > > > > > > major features we'd think about in a 'major
> release'
> > > sense
> > > > > > more
> > > > > > > > > > > > > approachable.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We should remove all deprecated things (as well as
> > > verify
> > > > > we
> > > > > > > > have the
> > > > > > > > > > > > > right list). We should remove/consider removal of
> > > > > deprecated
> > > > > > > > > > > > > concepts
> > > > > > > > > > > > > like templates. We should consider whether we can
> > > resolve
> > > > > the
> > > > > > > > > > > > > various
> > > > > > > > > > > > > ways we've handled what are now parameters down to
> > one
> > > > > clean
> > > > > > > > > > > > > approach.
> > > > > > > > > > > > > We should remove options in the nifi.properties
> which
> > > turn
> > > > > > out
> > > > > > > to
> > > > > > > > > > > > > never be used quite right (if there are). There is
> > > quite a
> > > > > > bit
> > > > > > > we
> > > > > > > > > > > > > can
> > > > > > > > > > > > > do purely in the name of tech debt reduction.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Lots to consider here but I think this is the right
> > > > > > discussion.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Than ks
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> > > > > > bbende@gmail.com
> > > > > > > > > > <mailto:
> > > > > > > > > > > > > bbende@gmail.com>>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > I'm a +1 for this... Not sure if this falls under
> > > "Removing
> > > > > > > > > > > > > Deprecated
> > > > > > > > > > > > > Components", but I think we should also look at
> > > anything
> > > > > that
> > > > > > > has
> > > > > > > > > > > > > been
> > > > > > > > > > > > > marked as deprecated throughout the code base as a
> > > > > candidate
> > > > > > > for
> > > > > > > > > > > > > removal. There are quite a few classes, methods,
> > > > > properties,
> > > > > > > etc
> > > > > > > > > > > > > that
> > > > > > > > > > > > > have been waiting for a chance to be removed.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > > > > > > > > > > <exceptionfactory@apache.org<mailto:
> > > > > > > exceptionfactory@apache.org
> > > > > > > > >>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > Team,
> > > > > > > > > > > > >
> > > > > > > > > > > > > With all of the excellent work that many have
> > > contributed
> > > > > to
> > > > > > > NiFi
> > > > > > > > > > > > > over the
> > > > > > > > > > > > > years, the code base has also accumulated some
> amount
> > > of
> > > > > > > > technical
> > > > > > > > > > > > > debt. A
> > > > > > > > > > > > > handful of components have been marked as
> deprecated,
> > > and
> > > > > > some
> > > > > > > > > > > > > components
> > > > > > > > > > > > > remain in the code base to support integration with
> > old
> > > > > > > versions
> > > > > > > > > > > > > of various
> > > > > > > > > > > > > products. Following the principles of semantic
> > > versioning,
> > > > > > > > > > > > > introducing a
> > > > > > > > > > > > > major release would provide the opportunity to
> remove
> > > these
> > > > > > > > > > > > > deprecated and
> > > > > > > > > > > > > unsupported components.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Rather than focusing the next major release on new
> > > > > features,
> > > > > > > what
> > > > > > > > > > > > > do you
> > > > > > > > > > > > > think about focusing on technical debt removal?
> This
> > > > > approach
> > > > > > > > > > > > > would not
> > > > > > > > > > > > > make for the most interesting release, but it
> > provides
> > > the
> > > > > > > > > > > > > opportunity to
> > > > > > > > > > > > > clean up elements that involve breaking changes.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Focusing on technical debt, at least three primary
> > > goals
> > > > > come
> > > > > > > to
> > > > > > > > > > > > > mind for
> > > > > > > > > > > > > the next major release:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Removal of deprecated and unmaintained
> components
> > > > > > > > > > > > > 2. Require Java 11 as the minimum supported version
> > > > > > > > > > > > > 3. Transition internal date and time handling to
> JSR
> > > 310
> > > > > > > > java.time
> > > > > > > > > > > > > components
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Removing Deprecated Components*
> > > > > > > > > > > > >
> > > > > > > > > > > > > Removing support for older and deprecated
> components
> > > > > > provides a
> > > > > > > > > > > > > great
> > > > > > > > > > > > > opportunity to improve the overall security posture
> > > when it
> > > > > > > comes
> > > > > > > > > > > > > to
> > > > > > > > > > > > > maintaining dependencies. The OWASP dependency
> plugin
> > > > > report
> > > > > > > > > > > > > currently
> > > > > > > > > > > > > generates 50 MB of HTML for questionable
> > dependencies,
> > > many
> > > > > > of
> > > > > > > > > > > > > which are
> > > > > > > > > > > > > related to old versions of various libraries.
> > > > > > > > > > > > >
> > > > > > > > > > > > > As a starting point, here are a handful of
> components
> > > and
> > > > > > > > > > > > > extension modules
> > > > > > > > > > > > > that could be targeted for removal in a major
> > version:
> > > > > > > > > > > > >
> > > > > > > > > > > > > - PostHTTP and GetHTTP
> > > > > > > > > > > > > - ListenLumberjack and the entire
> > > nifi-lumberjack-bundle
> > > > > > > > > > > > > - ListenBeats and the entire nifi-beats-bundle
> > > > > > > > > > > > > - Elasticsearch 5 components
> > > > > > > > > > > > > - Hive 1 and 2 components
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Requiring Java 11*
> > > > > > > > > > > > >
> > > > > > > > > > > > > Java 8 is now over seven years old, and NiFi has
> > > supported
> > > > > > > > general
> > > > > > > > > > > > > compatibility with Java 11 for several years. NiFi
> > > 1.14.0
> > > > > > > > > > > > > incorporated
> > > > > > > > > > > > > internal improvements specifically related to TLS
> > 1.3,
> > > > > which
> > > > > > > > > > > > > allowed
> > > > > > > > > > > > > closing out the long-running Java 11 compatibility
> > epic
> > > > > > > > NIFI-5174.
> > > > > > > > > > > > > Making
> > > > > > > > > > > > > Java 11 the minimum required version provides the
> > > > > opportunity
> > > > > > > to
> > > > > > > > > > > > > address
> > > > > > > > > > > > > any lingering edge cases and put NiFi in a better
> > > position
> > > > > to
> > > > > > > > > > > > > support
> > > > > > > > > > > > > current Java versions.
> > > > > > > > > > > > >
> > > > > > > > > > > > > *JSR 310 for Date and Time Handling*
> > > > > > > > > > > > >
> > > > > > > > > > > > > Without making the scope too broad, transitioning
> > > internal
> > > > > > date
> > > > > > > > > > > > > and time
> > > > > > > > > > > > > handling to use DateTimeFormatter instead of
> > > > > SimpleDateFormat
> > > > > > > > > > > > > would provide
> > > > > > > > > > > > > a number of advantages. The Java Time components
> > > provide
> > > > > much
> > > > > > > > > > > > > better
> > > > > > > > > > > > > clarity when it comes to handling localized date
> and
> > > time
> > > > > > > > > > > > > representations,
> > > > > > > > > > > > > and also avoid the inherent confusion of
> > java.sql.Date
> > > > > > > extending
> > > > > > > > > > > > > java.util.Date. Many internal components,
> > specifically
> > > > > > > > > > > > > Record-oriented
> > > > > > > > > > > > > processors and services, rely on date parsing,
> > leading
> > > to
> > > > > > > > > > > > > confusion and
> > > > > > > > > > > > > various workarounds. The pattern formats of
> > > > > SimpleDateFormat
> > > > > > > and
> > > > > > > > > > > > > DateTimeFormatter are very similar, but there are a
> > few
> > > > > > subtle
> > > > > > > > > > > > > differences.
> > > > > > > > > > > > > Making this transition would provide a much better
> > > > > foundation
> > > > > > > > > > > > > going forward.
> > > > > > > > > > > > > *Conclusion*
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for giving this proposal some consideration.
> > > Many of
> > > > > > you
> > > > > > > > > > > > > have been
> > > > > > > > > > > > > developing NiFi for years and I look forward to
> your
> > > > > > feedback.
> > > > > > > I
> > > > > > > > > > > > > would be
> > > > > > > > > > > > > glad to put together a more formalized
> recommendation
> > > on
> > > > > > > > > > > > > Confluence and
> > > > > > > > > > > > > write up Jira epics if this general approach sounds
> > > > > agreeable
> > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > community.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > David Handermann
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Adam Taft <ad...@adamtaft.com>.
I'm not seeing the side thread that was going to discuss deprecation of
PostHTTP. Has that thread started and I just don't see it?
One (significant?) concern with PostHTTP is the smooth integration of
NiFi-to-NiFi communication that is very transparently enabled with the
ListenHTTP and PostHTTP processors. There's some special logic in there for
handling flowfiles that InvokeHTTP doesn't really (nor should really) have.
I know of several (large) NiFi installations that rely on the PostHTTP /
ListenHTTP combination. It has enabled NiFi to NiFi communication for folks
reluctant or unable to enable site-to-site type configuration.
Honestly, I don't know why we'd want to "deprecate" any processor, as
opposed to just moving it to a new location. If these processors can be
ported and maintained to whatever the 2.0 API looks like, there's possibly
little harm keeping them around.
And by the way, what happened to the "marketplace" concept? Is this being
considered for 2.0 as well? Because relocating the deprecated processors
to an external nar might be the best solution. Losing PostHTTP entirely I
think would be a mistake, but I'd conceptually support relocating it.
Thanks,
/Adam
On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <jo...@gmail.com> wrote:
> Looks like we just need to knock out 5 JIRAs :) [1]
>
> I felt like we had a label folks were using at one point but quickly
> looking revealed nothing exciting. After this confluence page
> stabilizes a bit we can probably knock out some JIRAs and such.
>
> [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
>
> On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <ot...@gmail.com>
> wrote:
> >
> > I find myself wishing I had a list of all the jiras / issues that have
> > been put off for a 2.0 release because they required some change or
> another
> > :(
> >
> > From: Joe Witt <jo...@gmail.com> <jo...@gmail.com>
> > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > Date: July 27, 2021 at 12:30:35
> > To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> >
> > A few thoughts:
> >
> > 1. I would love to see deprecation notices show up in the UI in
> > various ways to help motivate users to move off things to more
> > supportable things. That is not a prerequisite for anything happening
> > however. Just a good feature/nice thing to do for users when someone
> > is able to tackle it.
> >
> > 2. The decision to deprecate something and to further remove it need
> > not mean there is a superior solution available. If that thing itself
> > isn't getting the love/attention it needs to be
> > maintained/supported/healthy going forward that alone is enough to
> > remove it. That might well be the case with PostHTTP [1] and for
> > comparison you can see how much effort has gone into InvokeHTTP [2].
> >
> > 3. When discussing a 2.0 release each thing we add as a 'must do' the
> > further away from reality such a release will become. We'll have to
> > get very specific about 'musts' vs 'wants'.
> >
> > [1]
> >
> https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
> > [2]
> >
> https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
> >
> > On Tue, Jul 27, 2021 at 8:47 AM David Handermann
> > <ex...@apache.org> wrote:
> > >
> > > Thanks Mark, providing a template or comparison statistics with Java
> > > versions and component configuration details would be very helpful. If
> it
> > > is possible to run tests using a public API or deployable service, that
> > > would also help confirm potential differences.
> > >
> > > Showing a deprecation notice in the UI could be helpful, perhaps as a
> > > configurable option. NIFI-8650 describes a general Flow Analysis
> > > capability, and it seems like that might be one possible way to surface
> > > deprecation warnings. For something more specific to component
> > deprecation,
> > > it seems important to find a balance between making it obvious and
> making
> > > it something that ends up getting ignored.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <ma...@gmail.com>
> wrote:
> > >
> > > > I'll start a new thread for PostHTTP when I get a template and/or
> > detailed
> > > > stats.
> > > >
> > > > I know the deprecation is noted in the documentation. That's a
> > necessary
> > > > and minimum level of notification. I was suggesting it be more
> obvious
> > in
> > > > the UI. I think it would be beneficial to somehow be aware of the
> > > > deprecation status simply by looking at the flow (perhaps even on the
> > > > summary pages too), and without having to open the documentation for
> > every
> > > > processor to confirm whether or not a component is marked as
> > deprecated.
> > > >
> > > > Thanks,
> > > > Mark
> > > >
> > > >
> > > > On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
> > > > exceptionfactory@apache.org> wrote:
> > > >
> > > > > Mark,
> > > > >
> > > > > Thanks for the feedback. It may be better to start a separate
> thread
> > on
> > > > > PostHTTP, but can you provide an example flow demonstrating the
> > > > performance
> > > > > differences between PostHTTP and InvokeHTTP?
> > > > >
> > > > > PostHTTP uses the Apache HttpComponents library, whereas InvokeHTTP
> > uses
> > > > > OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for
> > OkHttp,
> > > > > so it would be important to test with the most recent version. It
> is
> > also
> > > > > worth noting that test classes for PostHTTP are now integration
> tests
> > as
> > > > > opposed to unit tests, which means they are not executed as part of
> > the
> > > > > automated builds.
> > > > >
> > > > > The NiFi documentation includes the contents of the
> DeprecationNotice
> > for
> > > > > PostHTTP:
> > > > >
> > > > >
> > > > >
> > > >
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
> > > > >
> > > > > Regards,
> > > > > David Handermann
> > > > >
> > > > > On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <ma...@gmail.com>
> > wrote:
> > > > >
> > > > > > I'm strongly in favor of reducing tech debt, and moving
> > deliberately
> > > > to a
> > > > > > 2.0 release. I have a concern with just one processor that is
> > currently
> > > > > > marked as deprecated: PostHTTP. (I have not evaluated
> specifically
> > any
> > > > > > other deprecated components; I cannot say if there are or are not
> > > > similar
> > > > > > issues elsewhere.) I understand the rationale for deprecating
> this
> > > > > > processor in that it eliminates a processor whose functionality
> is
> > > > > > available elsewhere, specifically in the more flexible
> InvokeHTTP.
> > > > > However,
> > > > > > in my experience and testing, PostHTTP performs transfers with
> far
> > > > > greater
> > > > > > throughput than InvokeHTTP. I would not be in favor of removing
> > > > PostHTTP
> > > > > > unless/until InvokeHTTP is refactored to increase its throughput
> > > > > > capability.
> > > > > >
> > > > > > Has anyone else continued to use PostHTTP over InvokeHTTP for
> > similar
> > > > > > reasons? Or, is there a performance-related configuration for
> > > > InvokeHTTP
> > > > > I
> > > > > > may have missed?
> > > > > >
> > > > > > Also, in order to help facilitate a smooth transition to 2.0
> from a
> > > > user
> > > > > > perspective, would it be advisable to add some sort of visual
> > indicator
> > > > > in
> > > > > > the UI for components that are currently annotated as
> @Deprecated?
> > This
> > > > > > might help users proactively modify their flow prior to a release
> > that
> > > > > > would otherwise break it.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <bb...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > With the merging of MiNiFi and registry into the NiFi repo,
> we've
> > > > > > > actually gone more towards a mono-repo where everything is
> > released
> > > > > > > together. Eventually it would still be nice to have a smaller
> > base
> > > > > > > distribution containing just the framework and standard NARs,
> but
> > it
> > > > > > > is hard to tackle that until we provide a way for users to
> easily
> > > > > > > obtain the NARs in some other way.
> > > > > > >
> > > > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <
> > edward.armes@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Given the major version number shift and the spliting up of
> > > > > processors
> > > > > > > into
> > > > > > > > multiple NAR's. I'd like to suggest that we start
> individually
> > > > > > versioning
> > > > > > > > individual NARs/ bundles.
> > > > > > > >
> > > > > > > > I can see this bringing a large number of benifits including
> > making
> > > > > > Nifi
> > > > > > > > more deployable with things RPM, but also potentially
> allowing
> > > > those
> > > > > > that
> > > > > > > > have to deploy managed Nifi instances easier to upgrade.
> > > > > > > >
> > > > > > > > Edward
> > > > > > > >
> > > > > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, <
> ottobackwards@gmail.com>
> >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > The issue with updating the aws sdk is if it breaks any one
> > of
> > > > the
> > > > > > > > > processors.
> > > > > > > > > the Web Gateway API invoke processor for example is not
> using
> > a
> > > > > high
> > > > > > > level
> > > > > > > > > purpose build client and may break.
> > > > > > > > >
> > > > > > > > > If we change the aws version, we need to coordinate in
> such a
> > way
> > > > > > that
> > > > > > > they
> > > > > > > > > all
> > > > > > > > > can come along reasonably.
> > > > > > > > > IE: what happens if 1 or 2 break but the rest or OK?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > From: David Handermann <ex...@apache.org>
> > > > > > > > > <ex...@apache.org>
> > > > > > > > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > > > dev@nifi.apache.org>
> > > > > > > > > Date: July 26, 2021 at 09:33:42
> > > > > > > > > To: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > dev@nifi.apache.org
> > > > > >
> > > > > > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > > > > > > >
> > > > > > > > > Chris,
> > > > > > > > >
> > > > > > > > > Thanks for the reply and recommendations. It seems like
> some
> > of
> > > > the
> > > > > > > work to
> > > > > > > > > reorganize the module structure could be done outside of a
> > major
> > > > > > > release,
> > > > > > > > > but it would be great to target any breaking changes for
> 2.0.
> > > > > > Perhaps a
> > > > > > > > > separate feature proposal on module restructuring, with the
> > goal
> > > > of
> > > > > > > > > supporting optimized builds, would be a helpful way to move
> > that
> > > > > part
> > > > > > > of
> > > > > > > > > the discussion forward.
> > > > > > > > >
> > > > > > > > > Regarding updating AWS SDK to version 2, it seems like that
> > might
> > > > > be
> > > > > > > > > possible now. I haven't taken a close look at the
> referencing
> > > > > > > components,
> > > > > > > > > so I'm not sure about the level of effort involved. Minor
> > NiFi
> > > > > > version
> > > > > > > > > updates have incorporated major new versions of
> dependencies.
> > For
> > > > > > > example,
> > > > > > > > > NiFi 1.14 included an upgrade from Spring Framework 4 to 5.
> > On
> > > > the
> > > > > > one
> > > > > > > > > hand, including the AWS SDK update as part of a major
> release
> > > > seems
> > > > > > > > > helpful, but unless there are changes that break existing
> > > > component
> > > > > > > > > properties, upgrading the AWS SDK could be worked
> > independently.
> > > > > > > Others may
> > > > > > > > > have more insight into particular usage of that library.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > David Handermann
> > > > > > > > >
> > > > > > > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > > > > > > > > <ch...@naimuri.com.invalid> wrote:
> > > > > > > > >
> > > > > > > > > > Might be worth considering refactoring the build as part
> of
> > > > this
> > > > > > work
> > > > > > > > > too,
> > > > > > > > > > e.g. only building the bits of the repo affected by a
> > commit,
> > > > > etc.
> > > > > > -
> > > > > > > > > > discussed briefly in previous threads but don't think any
> > > > changes
> > > > > > > made
> > > > > > > > > yet.
> > > > > > > > > > If NARs/components are likely to be split up and
> refactored
> > > > then
> > > > > > such
> > > > > > > > > work
> > > > > > > > > > around the build probably makes sense to consider.
> > > > > > > > > >
> > > > > > > > > > I've a couple of PRs open that include updates to
> > Elasticsearch
> > > > > > > versions
> > > > > > > > > > already, although I stopped at 7.10.2 (after which
> Elastic
> > > > > changed
> > > > > > > > > licence)
> > > > > > > > > > in case there were licence concerns. But more work can be
> > done
> > > > to
> > > > > > > tidy up
> > > > > > > > > > the processors, absolutely.
> > > > > > > > > >
> > > > > > > > > > AWS libraries to v2 would seem a sensible move and a
> > refactor
> > > > of
> > > > > > > those
> > > > > > > > > > processors as well.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
> > > > > > > > > > Chris Sampson
> > > > > > > > > >
> > > > > > > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > > > > > > > > exceptionfactory@apache.org>
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for pointing out the standard NAR bundles, Mark.
> > There
> > > > > > are a
> > > > > > > > > > number
> > > > > > > > > > > of components in the standard NAR bundles with
> particular
> > > > > > > dependencies
> > > > > > > > > > that
> > > > > > > > > > > would make more sense in separate NARs. Reorganizing
> the
> > > > > standard
> > > > > > > NAR
> > > > > > > > > to
> > > > > > > > > > > components with limited dependencies and wide
> > applicability
> > > > > would
> > > > > > > > > > > definitely help with future maintenance.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > David Handermann
> > > > > > > > > > >
> > > > > > > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> > > > > > markap14@hotmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > There’s also some code that exists in order to
> maintain
> > > > > > backward
> > > > > > > > > > > > compatibility in the repositories. I would very much
> > like
> > > > the
> > > > > > > > > > > repositories
> > > > > > > > > > > > to contain no unnecessary code. And swap file format
> > > > supports
> > > > > > > really
> > > > > > > > > > old
> > > > > > > > > > > > formats. And the old impls of the repositories
> > themselves,
> > > > > like
> > > > > > > > > > > > PersistentProvRepo instead of WriteAheadProv Repo,
> etc.
> > > > Lots
> > > > > of
> > > > > > > stuff
> > > > > > > > > > > there
> > > > > > > > > > > > that can be removed. And some methods in
> ProcessSession
> > > > that
> > > > > > are
> > > > > > > > > never
> > > > > > > > > > > used
> > > > > > > > > > > > by any processor in the codebase but exists in the
> > public
> > > > API
> > > > > > so
> > > > > > > > > can’t
> > > > > > > > > > be
> > > > > > > > > > > > removed till 2.0.
> > > > > > > > > > > >
> > > > > > > > > > > > I think his is also a great time to clean up the
> > “standard
> > > > > > nar.”
> > > > > > > At
> > > > > > > > > > this
> > > > > > > > > > > > point, it’s something like 70 MB. And many of the
> > > > components
> > > > > > > there
> > > > > > > > > are
> > > > > > > > > > > not
> > > > > > > > > > > > really “standard” - things like connecting to FTP &
> > SFTP
> > > > > > > servers, XML
> > > > > > > > > > > > processing, Jolt transform, etc. could potentially be
> > moved
> > > > > > into
> > > > > > > > > other
> > > > > > > > > > > > nars. The
> > nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
> > > > is
> > > > > > > 6.9 MB
> > > > > > > > > is
> > > > > > > > > > > not
> > > > > > > > > > > > necessary for stateless or minifi java. Lots of
> things
> > > > > probably
> > > > > > > to
> > > > > > > > > > > > reconsider within the standard nar.
> > > > > > > > > > > >
> > > > > > > > > > > > I definitely think this is a reasonable approach, to
> > allow
> > > > > for
> > > > > > a
> > > > > > > 2.0
> > > > > > > > > > that
> > > > > > > > > > > > is not a huge feature release but allows the project
> to
> > be
> > > > > > > simpler
> > > > > > > > > and
> > > > > > > > > > > more
> > > > > > > > > > > > nimble.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > > -Mark
> > > > > > > > > > > >
> > > > > > > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > > > > > > mikerthomsen@gmail.com
> > > > > > > > > > > <mailto:
> > > > > > > > > > > > mikerthomsen@gmail.com>> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Russell,
> > > > > > > > > > > >
> > > > > > > > > > > > AFAICT from looking at Elastic's repos, the low level
> > REST
> > > > > > > client is
> > > > > > > > > > > > still fine.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > > > > > > > > > >
> > > > > > > > > > > > Our Elasticsearch support is spread over two NARs at
> > > > present.
> > > > > > One
> > > > > > > > > uses
> > > > > > > > > > > > OkHttp and the other uses that low level Elastic REST
> > > > client.
> > > > > > > > > > > > Therefore, I think we're fine on licensing for the
> > moment.
> > > > > > > > > > > >
> > > > > > > > > > > > Mike
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > > > > > > > > russ@windofkeltia.com
> > > > > > > > > > > > <ma...@windofkeltia.com>> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Bringing up Elastic also reminds me that the Elastic
> > > > > framework
> > > > > > > has
> > > > > > > > > just
> > > > > > > > > > > > recently transitioned out of Open Source, so to
> > acknowledge
> > > > > > that,
> > > > > > > > > maybe
> > > > > > > > > > > > some effort toward OpenSearch--I say this not
> > understanding
> > > > > > > exactly
> > > > > > > > > how
> > > > > > > > > > > > this sort of thing is considered in a large-scale,
> > > > > world-class
> > > > > > > > > software
> > > > > > > > > > > > project like Apache NiFi. (I'm not a contributor,
> just
> > a
> > > > > > grateful
> > > > > > > > > > > > consumer.)
> > > > > > > > > > > >
> > > > > > > > > > > > Russ
> > > > > > > > > > > >
> > > > > > > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > > > > > > > > > > Along with the itemized list for ancient components
> we
> > > > should
> > > > > > > look at
> > > > > > > > > > > > updating versions of drivers, SDKs, etc. for external
> > > > systems
> > > > > > > such as
> > > > > > > > > > > > Elasticsearch, Cassandra, etc. There may be breaking
> > > > changes
> > > > > > but
> > > > > > > 2.0
> > > > > > > > > > > > is probably the right time to get things up to date
> to
> > make
> > > > > > them
> > > > > > > more
> > > > > > > > > > > > useful to more people.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > > > > > > thenatog@gmail.com
> > > > > > > > > > > <mailto:
> > > > > > > > > > > > thenatog@gmail.com>> wrote:
> > > > > > > > > > > > I'm a +1 for removing pretty much all of this stuff.
> > There
> > > > > are
> > > > > > > > > security
> > > > > > > > > > > > implications to keeping old dependencies around, so
> the
> > > > more
> > > > > > old
> > > > > > > code
> > > > > > > > > > we
> > > > > > > > > > > > can remove the better. I agree that eventually we
> need
> > to
> > > > > move
> > > > > > to
> > > > > > > > > > > > supporting only Java 11+, and as our next release
> will
> > > > > probably
> > > > > > > be
> > > > > > > > > > about
> > > > > > > > > > > 4
> > > > > > > > > > > > - 6 months from now that doesn't seem too soon. We
> > could
> > > > > > > potentially
> > > > > > > > > > > break
> > > > > > > > > > > > this in two and remove the deprecated processors and
> > leave
> > > > > 1.x
> > > > > > on
> > > > > > > > > Java
> > > > > > > > > > 8,
> > > > > > > > > > > > and finally start on 2.x which would support only
> Java
> > 11.
> > > > > I'm
> > > > > > > unsure
> > > > > > > > > > of
> > > > > > > > > > > > what implications changing the date and time handling
> > would
> > > > > > have
> > > > > > > -
> > > > > > > > > for
> > > > > > > > > > > > running systems that use long term historical logs,
> > > > > unexpected
> > > > > > > > > impacts
> > > > > > > > > > to
> > > > > > > > > > > > time logging could be a problem.
> > > > > > > > > > > >
> > > > > > > > > > > > As Joe says I think feature work will have to be
> > dedicated
> > > > to
> > > > > > > 2.x and
> > > > > > > > > > we
> > > > > > > > > > > > could support 1.x for security fixes for some period
> of
> > > > time.
> > > > > > 2.x
> > > > > > > > > seems
> > > > > > > > > > > > like a gargantuan task but it's probably time to get
> > > > started.
> > > > > > Not
> > > > > > > > > sure
> > > > > > > > > > > how
> > > > > > > > > > > > we handle all open PRs and the transition between 1.x
> > and
> > > > > 2.x.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> > > > > joe.witt@gmail.com
> > > > > > > > > <mailto:
> > > > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Jon
> > > > > > > > > > > >
> > > > > > > > > > > > You're right we have to be careful and you're right
> > there
> > > > are
> > > > > > > still
> > > > > > > > > > > > significant Java 8 users out there. But we also have
> to
> > be
> > > > > > > careful
> > > > > > > > > > > > about security and sustainability of the codebase. If
> > we
> > > > had
> > > > > > > talked
> > > > > > > > > > > > about this last year when that article came out I'd
> > have
> > > > > agreed
> > > > > > > it is
> > > > > > > > > > > > too early. Interestingly that link seems to get
> updated
> > > > and I
> > > > > > > tried
> > > > > > > > > > > > [1] and found more recent data (not sure how recent).
> > > > Anyway
> > > > > it
> > > > > > > > > > > > suggests Java 8 is still the top dog but we see good
> > growth
> > > > > on
> > > > > > > 11. In
> > > > > > > > > > > > my $dayjob this aligns to what I'm seeing too.
> > Customers
> > > > > didn't
> > > > > > > seem
> > > > > > > > > > > > to care about Java 11 until later half last year and
> > now
> > > > > > > suddenly it
> > > > > > > > > > > > is all over the place.
> > > > > > > > > > > >
> > > > > > > > > > > > I think once we put out a NiFi 2.0 release we'd see
> > rapid
> > > > > > > decrease in
> > > > > > > > > > > > work on the 1.x line just being blunt. We did this
> many
> > > > years
> > > > > > ago
> > > > > > > > > > > > with 0.x to 1.x and we stood behind 0.x for a while
> > (maybe
> > > > a
> > > > > > > year or
> > > > > > > > > > > > so) but it was purely bug fix/security related bits.
> We
> > > > would
> > > > > > > need to
> > > > > > > > > > > > do something similar. But feature work would almost
> > > > certainly
> > > > > > go
> > > > > > > to
> > > > > > > > > > > > the 2.x line. Maybe there are other workable models
> but
> > my
> > > > > > > instinct
> > > > > > > > > > > > suggests this is likely to follow a similar path.
> > > > > > > > > > > >
> > > > > > > > > > > > ...anyway I agree it isn't that easy of a call to
> dump
> > Java
> > > > > 8.
> > > > > > We
> > > > > > > > > > > > need to make the call in both the interests of the
> user
> > > > base
> > > > > > and
> > > > > > > the
> > > > > > > > > > > > contributor base of the community.
> > > > > > > > > > > >
> > > > > > > > > > > > [1]
> https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > > Joe
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
> > > > joe.witt@gmail.com
> > > > > > > <mailto:
> > > > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > > > > Russ
> > > > > > > > > > > >
> > > > > > > > > > > > Yeah the flow registry is a key part of it. But also
> > now
> > > > you
> > > > > > can
> > > > > > > > > > > > download the flow definition in JSON (upload i think
> is
> > > > there
> > > > > > now
> > > > > > > > > > > > too). Templates offered a series of challenges such
> as
> > we
> > > > > store
> > > > > > > them
> > > > > > > > > > > > in the flow definition which has made flows massive
> in
> > an
> > > > > > > unintended
> > > > > > > > > > > > way which isn't fun for cluster behavior.
> > > > > > > > > > > >
> > > > > > > > > > > > We have a couple cases where we headed down a
> > particular
> > > > > > concept
> > > > > > > and
> > > > > > > > > > > > came up with better approaches later. We need to
> > reconcile
> > > > > > these
> > > > > > > with
> > > > > > > > > > > > the benefit of hindsight, and while being careful to
> be
> > not
> > > > > > > overly
> > > > > > > > > > > > disruptive to existing users, to reduce the
> > > > > > codebase/maintenance
> > > > > > > > > > > > burden and allow continued evolution of the project.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > > > > > > > > russ@windofkeltia.com
> > > > > > > > > > > > <ma...@windofkeltia.com>>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > Joe,
> > > > > > > > > > > >
> > > > > > > > > > > > I apologize for the off-topic intrusion, but what
> > replaces
> > > > > > > templates?
> > > > > > > > > > > > The Registry? Templates rocked and we have used them
> > since
> > > > > > 0.5.x.
> > > > > > > > > > > >
> > > > > > > > > > > > Russ
> > > > > > > > > > > >
> > > > > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > > > > > > > > David,
> > > > > > > > > > > >
> > > > > > > > > > > > I think this is a highly reasonable approach and
> such a
> > > > focus
> > > > > > > will
> > > > > > > > > > > > greatly help make a 2.0 release far more approachable
> > to
> > > > > knock
> > > > > > > out.
> > > > > > > > > > > > Not only that but tech debt reduction would help make
> > work
> > > > > > > towards
> > > > > > > > > > > > major features we'd think about in a 'major release'
> > sense
> > > > > more
> > > > > > > > > > > > approachable.
> > > > > > > > > > > >
> > > > > > > > > > > > We should remove all deprecated things (as well as
> > verify
> > > > we
> > > > > > > have the
> > > > > > > > > > > > right list). We should remove/consider removal of
> > > > deprecated
> > > > > > > > > > > > concepts
> > > > > > > > > > > > like templates. We should consider whether we can
> > resolve
> > > > the
> > > > > > > > > > > > various
> > > > > > > > > > > > ways we've handled what are now parameters down to
> one
> > > > clean
> > > > > > > > > > > > approach.
> > > > > > > > > > > > We should remove options in the nifi.properties which
> > turn
> > > > > out
> > > > > > to
> > > > > > > > > > > > never be used quite right (if there are). There is
> > quite a
> > > > > bit
> > > > > > we
> > > > > > > > > > > > can
> > > > > > > > > > > > do purely in the name of tech debt reduction.
> > > > > > > > > > > >
> > > > > > > > > > > > Lots to consider here but I think this is the right
> > > > > discussion.
> > > > > > > > > > > >
> > > > > > > > > > > > Than ks
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> > > > > bbende@gmail.com
> > > > > > > > > <mailto:
> > > > > > > > > > > > bbende@gmail.com>>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > I'm a +1 for this... Not sure if this falls under
> > "Removing
> > > > > > > > > > > > Deprecated
> > > > > > > > > > > > Components", but I think we should also look at
> > anything
> > > > that
> > > > > > has
> > > > > > > > > > > > been
> > > > > > > > > > > > marked as deprecated throughout the code base as a
> > > > candidate
> > > > > > for
> > > > > > > > > > > > removal. There are quite a few classes, methods,
> > > > properties,
> > > > > > etc
> > > > > > > > > > > > that
> > > > > > > > > > > > have been waiting for a chance to be removed.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > > > > > > > > > <exceptionfactory@apache.org<mailto:
> > > > > > exceptionfactory@apache.org
> > > > > > > >>
> > > > > > > > > > wrote:
> > > > > > > > > > > > Team,
> > > > > > > > > > > >
> > > > > > > > > > > > With all of the excellent work that many have
> > contributed
> > > > to
> > > > > > NiFi
> > > > > > > > > > > > over the
> > > > > > > > > > > > years, the code base has also accumulated some amount
> > of
> > > > > > > technical
> > > > > > > > > > > > debt. A
> > > > > > > > > > > > handful of components have been marked as deprecated,
> > and
> > > > > some
> > > > > > > > > > > > components
> > > > > > > > > > > > remain in the code base to support integration with
> old
> > > > > > versions
> > > > > > > > > > > > of various
> > > > > > > > > > > > products. Following the principles of semantic
> > versioning,
> > > > > > > > > > > > introducing a
> > > > > > > > > > > > major release would provide the opportunity to remove
> > these
> > > > > > > > > > > > deprecated and
> > > > > > > > > > > > unsupported components.
> > > > > > > > > > > >
> > > > > > > > > > > > Rather than focusing the next major release on new
> > > > features,
> > > > > > what
> > > > > > > > > > > > do you
> > > > > > > > > > > > think about focusing on technical debt removal? This
> > > > approach
> > > > > > > > > > > > would not
> > > > > > > > > > > > make for the most interesting release, but it
> provides
> > the
> > > > > > > > > > > > opportunity to
> > > > > > > > > > > > clean up elements that involve breaking changes.
> > > > > > > > > > > >
> > > > > > > > > > > > Focusing on technical debt, at least three primary
> > goals
> > > > come
> > > > > > to
> > > > > > > > > > > > mind for
> > > > > > > > > > > > the next major release:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Removal of deprecated and unmaintained components
> > > > > > > > > > > > 2. Require Java 11 as the minimum supported version
> > > > > > > > > > > > 3. Transition internal date and time handling to JSR
> > 310
> > > > > > > java.time
> > > > > > > > > > > > components
> > > > > > > > > > > >
> > > > > > > > > > > > *Removing Deprecated Components*
> > > > > > > > > > > >
> > > > > > > > > > > > Removing support for older and deprecated components
> > > > > provides a
> > > > > > > > > > > > great
> > > > > > > > > > > > opportunity to improve the overall security posture
> > when it
> > > > > > comes
> > > > > > > > > > > > to
> > > > > > > > > > > > maintaining dependencies. The OWASP dependency plugin
> > > > report
> > > > > > > > > > > > currently
> > > > > > > > > > > > generates 50 MB of HTML for questionable
> dependencies,
> > many
> > > > > of
> > > > > > > > > > > > which are
> > > > > > > > > > > > related to old versions of various libraries.
> > > > > > > > > > > >
> > > > > > > > > > > > As a starting point, here are a handful of components
> > and
> > > > > > > > > > > > extension modules
> > > > > > > > > > > > that could be targeted for removal in a major
> version:
> > > > > > > > > > > >
> > > > > > > > > > > > - PostHTTP and GetHTTP
> > > > > > > > > > > > - ListenLumberjack and the entire
> > nifi-lumberjack-bundle
> > > > > > > > > > > > - ListenBeats and the entire nifi-beats-bundle
> > > > > > > > > > > > - Elasticsearch 5 components
> > > > > > > > > > > > - Hive 1 and 2 components
> > > > > > > > > > > >
> > > > > > > > > > > > *Requiring Java 11*
> > > > > > > > > > > >
> > > > > > > > > > > > Java 8 is now over seven years old, and NiFi has
> > supported
> > > > > > > general
> > > > > > > > > > > > compatibility with Java 11 for several years. NiFi
> > 1.14.0
> > > > > > > > > > > > incorporated
> > > > > > > > > > > > internal improvements specifically related to TLS
> 1.3,
> > > > which
> > > > > > > > > > > > allowed
> > > > > > > > > > > > closing out the long-running Java 11 compatibility
> epic
> > > > > > > NIFI-5174.
> > > > > > > > > > > > Making
> > > > > > > > > > > > Java 11 the minimum required version provides the
> > > > opportunity
> > > > > > to
> > > > > > > > > > > > address
> > > > > > > > > > > > any lingering edge cases and put NiFi in a better
> > position
> > > > to
> > > > > > > > > > > > support
> > > > > > > > > > > > current Java versions.
> > > > > > > > > > > >
> > > > > > > > > > > > *JSR 310 for Date and Time Handling*
> > > > > > > > > > > >
> > > > > > > > > > > > Without making the scope too broad, transitioning
> > internal
> > > > > date
> > > > > > > > > > > > and time
> > > > > > > > > > > > handling to use DateTimeFormatter instead of
> > > > SimpleDateFormat
> > > > > > > > > > > > would provide
> > > > > > > > > > > > a number of advantages. The Java Time components
> > provide
> > > > much
> > > > > > > > > > > > better
> > > > > > > > > > > > clarity when it comes to handling localized date and
> > time
> > > > > > > > > > > > representations,
> > > > > > > > > > > > and also avoid the inherent confusion of
> java.sql.Date
> > > > > > extending
> > > > > > > > > > > > java.util.Date. Many internal components,
> specifically
> > > > > > > > > > > > Record-oriented
> > > > > > > > > > > > processors and services, rely on date parsing,
> leading
> > to
> > > > > > > > > > > > confusion and
> > > > > > > > > > > > various workarounds. The pattern formats of
> > > > SimpleDateFormat
> > > > > > and
> > > > > > > > > > > > DateTimeFormatter are very similar, but there are a
> few
> > > > > subtle
> > > > > > > > > > > > differences.
> > > > > > > > > > > > Making this transition would provide a much better
> > > > foundation
> > > > > > > > > > > > going forward.
> > > > > > > > > > > > *Conclusion*
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for giving this proposal some consideration.
> > Many of
> > > > > you
> > > > > > > > > > > > have been
> > > > > > > > > > > > developing NiFi for years and I look forward to your
> > > > > feedback.
> > > > > > I
> > > > > > > > > > > > would be
> > > > > > > > > > > > glad to put together a more formalized recommendation
> > on
> > > > > > > > > > > > Confluence and
> > > > > > > > > > > > write up Jira epics if this general approach sounds
> > > > agreeable
> > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > community.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > David Handermann
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Joe Witt <jo...@gmail.com>.
Looks like we just need to knock out 5 JIRAs :) [1]
I felt like we had a label folks were using at one point but quickly
looking revealed nothing exciting. After this confluence page
stabilizes a bit we can probably knock out some JIRAs and such.
[1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <ot...@gmail.com> wrote:
>
> I find myself wishing I had a list of all the jiras / issues that have
> been put off for a 2.0 release because they required some change or another
> :(
>
> From: Joe Witt <jo...@gmail.com> <jo...@gmail.com>
> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> Date: July 27, 2021 at 12:30:35
> To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
>
> A few thoughts:
>
> 1. I would love to see deprecation notices show up in the UI in
> various ways to help motivate users to move off things to more
> supportable things. That is not a prerequisite for anything happening
> however. Just a good feature/nice thing to do for users when someone
> is able to tackle it.
>
> 2. The decision to deprecate something and to further remove it need
> not mean there is a superior solution available. If that thing itself
> isn't getting the love/attention it needs to be
> maintained/supported/healthy going forward that alone is enough to
> remove it. That might well be the case with PostHTTP [1] and for
> comparison you can see how much effort has gone into InvokeHTTP [2].
>
> 3. When discussing a 2.0 release each thing we add as a 'must do' the
> further away from reality such a release will become. We'll have to
> get very specific about 'musts' vs 'wants'.
>
> [1]
> https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
> [2]
> https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
>
> On Tue, Jul 27, 2021 at 8:47 AM David Handermann
> <ex...@apache.org> wrote:
> >
> > Thanks Mark, providing a template or comparison statistics with Java
> > versions and component configuration details would be very helpful. If it
> > is possible to run tests using a public API or deployable service, that
> > would also help confirm potential differences.
> >
> > Showing a deprecation notice in the UI could be helpful, perhaps as a
> > configurable option. NIFI-8650 describes a general Flow Analysis
> > capability, and it seems like that might be one possible way to surface
> > deprecation warnings. For something more specific to component
> deprecation,
> > it seems important to find a balance between making it obvious and making
> > it something that ends up getting ignored.
> >
> > Regards,
> > David Handermann
> >
> > On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <ma...@gmail.com> wrote:
> >
> > > I'll start a new thread for PostHTTP when I get a template and/or
> detailed
> > > stats.
> > >
> > > I know the deprecation is noted in the documentation. That's a
> necessary
> > > and minimum level of notification. I was suggesting it be more obvious
> in
> > > the UI. I think it would be beneficial to somehow be aware of the
> > > deprecation status simply by looking at the flow (perhaps even on the
> > > summary pages too), and without having to open the documentation for
> every
> > > processor to confirm whether or not a component is marked as
> deprecated.
> > >
> > > Thanks,
> > > Mark
> > >
> > >
> > > On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
> > > exceptionfactory@apache.org> wrote:
> > >
> > > > Mark,
> > > >
> > > > Thanks for the feedback. It may be better to start a separate thread
> on
> > > > PostHTTP, but can you provide an example flow demonstrating the
> > > performance
> > > > differences between PostHTTP and InvokeHTTP?
> > > >
> > > > PostHTTP uses the Apache HttpComponents library, whereas InvokeHTTP
> uses
> > > > OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for
> OkHttp,
> > > > so it would be important to test with the most recent version. It is
> also
> > > > worth noting that test classes for PostHTTP are now integration tests
> as
> > > > opposed to unit tests, which means they are not executed as part of
> the
> > > > automated builds.
> > > >
> > > > The NiFi documentation includes the contents of the DeprecationNotice
> for
> > > > PostHTTP:
> > > >
> > > >
> > > >
> > >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <ma...@gmail.com>
> wrote:
> > > >
> > > > > I'm strongly in favor of reducing tech debt, and moving
> deliberately
> > > to a
> > > > > 2.0 release. I have a concern with just one processor that is
> currently
> > > > > marked as deprecated: PostHTTP. (I have not evaluated specifically
> any
> > > > > other deprecated components; I cannot say if there are or are not
> > > similar
> > > > > issues elsewhere.) I understand the rationale for deprecating this
> > > > > processor in that it eliminates a processor whose functionality is
> > > > > available elsewhere, specifically in the more flexible InvokeHTTP.
> > > > However,
> > > > > in my experience and testing, PostHTTP performs transfers with far
> > > > greater
> > > > > throughput than InvokeHTTP. I would not be in favor of removing
> > > PostHTTP
> > > > > unless/until InvokeHTTP is refactored to increase its throughput
> > > > > capability.
> > > > >
> > > > > Has anyone else continued to use PostHTTP over InvokeHTTP for
> similar
> > > > > reasons? Or, is there a performance-related configuration for
> > > InvokeHTTP
> > > > I
> > > > > may have missed?
> > > > >
> > > > > Also, in order to help facilitate a smooth transition to 2.0 from a
> > > user
> > > > > perspective, would it be advisable to add some sort of visual
> indicator
> > > > in
> > > > > the UI for components that are currently annotated as @Deprecated?
> This
> > > > > might help users proactively modify their flow prior to a release
> that
> > > > > would otherwise break it.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <bb...@gmail.com>
> wrote:
> > > > >
> > > > > > With the merging of MiNiFi and registry into the NiFi repo, we've
> > > > > > actually gone more towards a mono-repo where everything is
> released
> > > > > > together. Eventually it would still be nice to have a smaller
> base
> > > > > > distribution containing just the framework and standard NARs, but
> it
> > > > > > is hard to tackle that until we provide a way for users to easily
> > > > > > obtain the NARs in some other way.
> > > > > >
> > > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <
> edward.armes@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Given the major version number shift and the spliting up of
> > > > processors
> > > > > > into
> > > > > > > multiple NAR's. I'd like to suggest that we start individually
> > > > > versioning
> > > > > > > individual NARs/ bundles.
> > > > > > >
> > > > > > > I can see this bringing a large number of benifits including
> making
> > > > > Nifi
> > > > > > > more deployable with things RPM, but also potentially allowing
> > > those
> > > > > that
> > > > > > > have to deploy managed Nifi instances easier to upgrade.
> > > > > > >
> > > > > > > Edward
> > > > > > >
> > > > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, <ot...@gmail.com>
>
> > > > > wrote:
> > > > > > >
> > > > > > > > The issue with updating the aws sdk is if it breaks any one
> of
> > > the
> > > > > > > > processors.
> > > > > > > > the Web Gateway API invoke processor for example is not using
> a
> > > > high
> > > > > > level
> > > > > > > > purpose build client and may break.
> > > > > > > >
> > > > > > > > If we change the aws version, we need to coordinate in such a
> way
> > > > > that
> > > > > > they
> > > > > > > > all
> > > > > > > > can come along reasonably.
> > > > > > > > IE: what happens if 1 or 2 break but the rest or OK?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From: David Handermann <ex...@apache.org>
> > > > > > > > <ex...@apache.org>
> > > > > > > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > > dev@nifi.apache.org>
> > > > > > > > Date: July 26, 2021 at 09:33:42
> > > > > > > > To: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > dev@nifi.apache.org
> > > > >
> > > > > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > > > > > >
> > > > > > > > Chris,
> > > > > > > >
> > > > > > > > Thanks for the reply and recommendations. It seems like some
> of
> > > the
> > > > > > work to
> > > > > > > > reorganize the module structure could be done outside of a
> major
> > > > > > release,
> > > > > > > > but it would be great to target any breaking changes for 2.0.
> > > > > Perhaps a
> > > > > > > > separate feature proposal on module restructuring, with the
> goal
> > > of
> > > > > > > > supporting optimized builds, would be a helpful way to move
> that
> > > > part
> > > > > > of
> > > > > > > > the discussion forward.
> > > > > > > >
> > > > > > > > Regarding updating AWS SDK to version 2, it seems like that
> might
> > > > be
> > > > > > > > possible now. I haven't taken a close look at the referencing
> > > > > > components,
> > > > > > > > so I'm not sure about the level of effort involved. Minor
> NiFi
> > > > > version
> > > > > > > > updates have incorporated major new versions of dependencies.
> For
> > > > > > example,
> > > > > > > > NiFi 1.14 included an upgrade from Spring Framework 4 to 5.
> On
> > > the
> > > > > one
> > > > > > > > hand, including the AWS SDK update as part of a major release
> > > seems
> > > > > > > > helpful, but unless there are changes that break existing
> > > component
> > > > > > > > properties, upgrading the AWS SDK could be worked
> independently.
> > > > > > Others may
> > > > > > > > have more insight into particular usage of that library.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > David Handermann
> > > > > > > >
> > > > > > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > > > > > > > <ch...@naimuri.com.invalid> wrote:
> > > > > > > >
> > > > > > > > > Might be worth considering refactoring the build as part of
> > > this
> > > > > work
> > > > > > > > too,
> > > > > > > > > e.g. only building the bits of the repo affected by a
> commit,
> > > > etc.
> > > > > -
> > > > > > > > > discussed briefly in previous threads but don't think any
> > > changes
> > > > > > made
> > > > > > > > yet.
> > > > > > > > > If NARs/components are likely to be split up and refactored
> > > then
> > > > > such
> > > > > > > > work
> > > > > > > > > around the build probably makes sense to consider.
> > > > > > > > >
> > > > > > > > > I've a couple of PRs open that include updates to
> Elasticsearch
> > > > > > versions
> > > > > > > > > already, although I stopped at 7.10.2 (after which Elastic
> > > > changed
> > > > > > > > licence)
> > > > > > > > > in case there were licence concerns. But more work can be
> done
> > > to
> > > > > > tidy up
> > > > > > > > > the processors, absolutely.
> > > > > > > > >
> > > > > > > > > AWS libraries to v2 would seem a sensible move and a
> refactor
> > > of
> > > > > > those
> > > > > > > > > processors as well.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris Sampson
> > > > > > > > >
> > > > > > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > > > > > > > exceptionfactory@apache.org>
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks for pointing out the standard NAR bundles, Mark.
> There
> > > > > are a
> > > > > > > > > number
> > > > > > > > > > of components in the standard NAR bundles with particular
> > > > > > dependencies
> > > > > > > > > that
> > > > > > > > > > would make more sense in separate NARs. Reorganizing the
> > > > standard
> > > > > > NAR
> > > > > > > > to
> > > > > > > > > > components with limited dependencies and wide
> applicability
> > > > would
> > > > > > > > > > definitely help with future maintenance.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > David Handermann
> > > > > > > > > >
> > > > > > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> > > > > markap14@hotmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > There’s also some code that exists in order to maintain
> > > > > backward
> > > > > > > > > > > compatibility in the repositories. I would very much
> like
> > > the
> > > > > > > > > > repositories
> > > > > > > > > > > to contain no unnecessary code. And swap file format
> > > supports
> > > > > > really
> > > > > > > > > old
> > > > > > > > > > > formats. And the old impls of the repositories
> themselves,
> > > > like
> > > > > > > > > > > PersistentProvRepo instead of WriteAheadProv Repo, etc.
> > > Lots
> > > > of
> > > > > > stuff
> > > > > > > > > > there
> > > > > > > > > > > that can be removed. And some methods in ProcessSession
> > > that
> > > > > are
> > > > > > > > never
> > > > > > > > > > used
> > > > > > > > > > > by any processor in the codebase but exists in the
> public
> > > API
> > > > > so
> > > > > > > > can’t
> > > > > > > > > be
> > > > > > > > > > > removed till 2.0.
> > > > > > > > > > >
> > > > > > > > > > > I think his is also a great time to clean up the
> “standard
> > > > > nar.”
> > > > > > At
> > > > > > > > > this
> > > > > > > > > > > point, it’s something like 70 MB. And many of the
> > > components
> > > > > > there
> > > > > > > > are
> > > > > > > > > > not
> > > > > > > > > > > really “standard” - things like connecting to FTP &
> SFTP
> > > > > > servers, XML
> > > > > > > > > > > processing, Jolt transform, etc. could potentially be
> moved
> > > > > into
> > > > > > > > other
> > > > > > > > > > > nars. The
> nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
> > > is
> > > > > > 6.9 MB
> > > > > > > > is
> > > > > > > > > > not
> > > > > > > > > > > necessary for stateless or minifi java. Lots of things
> > > > probably
> > > > > > to
> > > > > > > > > > > reconsider within the standard nar.
> > > > > > > > > > >
> > > > > > > > > > > I definitely think this is a reasonable approach, to
> allow
> > > > for
> > > > > a
> > > > > > 2.0
> > > > > > > > > that
> > > > > > > > > > > is not a huge feature release but allows the project to
> be
> > > > > > simpler
> > > > > > > > and
> > > > > > > > > > more
> > > > > > > > > > > nimble.
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > > -Mark
> > > > > > > > > > >
> > > > > > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > > > > > mikerthomsen@gmail.com
> > > > > > > > > > <mailto:
> > > > > > > > > > > mikerthomsen@gmail.com>> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Russell,
> > > > > > > > > > >
> > > > > > > > > > > AFAICT from looking at Elastic's repos, the low level
> REST
> > > > > > client is
> > > > > > > > > > > still fine.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > > > > > > > > >
> > > > > > > > > > > Our Elasticsearch support is spread over two NARs at
> > > present.
> > > > > One
> > > > > > > > uses
> > > > > > > > > > > OkHttp and the other uses that low level Elastic REST
> > > client.
> > > > > > > > > > > Therefore, I think we're fine on licensing for the
> moment.
> > > > > > > > > > >
> > > > > > > > > > > Mike
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > > > > > > > russ@windofkeltia.com
> > > > > > > > > > > <ma...@windofkeltia.com>> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Bringing up Elastic also reminds me that the Elastic
> > > > framework
> > > > > > has
> > > > > > > > just
> > > > > > > > > > > recently transitioned out of Open Source, so to
> acknowledge
> > > > > that,
> > > > > > > > maybe
> > > > > > > > > > > some effort toward OpenSearch--I say this not
> understanding
> > > > > > exactly
> > > > > > > > how
> > > > > > > > > > > this sort of thing is considered in a large-scale,
> > > > world-class
> > > > > > > > software
> > > > > > > > > > > project like Apache NiFi. (I'm not a contributor, just
> a
> > > > > grateful
> > > > > > > > > > > consumer.)
> > > > > > > > > > >
> > > > > > > > > > > Russ
> > > > > > > > > > >
> > > > > > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > > > > > > > > > Along with the itemized list for ancient components we
> > > should
> > > > > > look at
> > > > > > > > > > > updating versions of drivers, SDKs, etc. for external
> > > systems
> > > > > > such as
> > > > > > > > > > > Elasticsearch, Cassandra, etc. There may be breaking
> > > changes
> > > > > but
> > > > > > 2.0
> > > > > > > > > > > is probably the right time to get things up to date to
> make
> > > > > them
> > > > > > more
> > > > > > > > > > > useful to more people.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > > > > > thenatog@gmail.com
> > > > > > > > > > <mailto:
> > > > > > > > > > > thenatog@gmail.com>> wrote:
> > > > > > > > > > > I'm a +1 for removing pretty much all of this stuff.
> There
> > > > are
> > > > > > > > security
> > > > > > > > > > > implications to keeping old dependencies around, so the
> > > more
> > > > > old
> > > > > > code
> > > > > > > > > we
> > > > > > > > > > > can remove the better. I agree that eventually we need
> to
> > > > move
> > > > > to
> > > > > > > > > > > supporting only Java 11+, and as our next release will
> > > > probably
> > > > > > be
> > > > > > > > > about
> > > > > > > > > > 4
> > > > > > > > > > > - 6 months from now that doesn't seem too soon. We
> could
> > > > > > potentially
> > > > > > > > > > break
> > > > > > > > > > > this in two and remove the deprecated processors and
> leave
> > > > 1.x
> > > > > on
> > > > > > > > Java
> > > > > > > > > 8,
> > > > > > > > > > > and finally start on 2.x which would support only Java
> 11.
> > > > I'm
> > > > > > unsure
> > > > > > > > > of
> > > > > > > > > > > what implications changing the date and time handling
> would
> > > > > have
> > > > > > -
> > > > > > > > for
> > > > > > > > > > > running systems that use long term historical logs,
> > > > unexpected
> > > > > > > > impacts
> > > > > > > > > to
> > > > > > > > > > > time logging could be a problem.
> > > > > > > > > > >
> > > > > > > > > > > As Joe says I think feature work will have to be
> dedicated
> > > to
> > > > > > 2.x and
> > > > > > > > > we
> > > > > > > > > > > could support 1.x for security fixes for some period of
> > > time.
> > > > > 2.x
> > > > > > > > seems
> > > > > > > > > > > like a gargantuan task but it's probably time to get
> > > started.
> > > > > Not
> > > > > > > > sure
> > > > > > > > > > how
> > > > > > > > > > > we handle all open PRs and the transition between 1.x
> and
> > > > 2.x.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> > > > joe.witt@gmail.com
> > > > > > > > <mailto:
> > > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Jon
> > > > > > > > > > >
> > > > > > > > > > > You're right we have to be careful and you're right
> there
> > > are
> > > > > > still
> > > > > > > > > > > significant Java 8 users out there. But we also have to
> be
> > > > > > careful
> > > > > > > > > > > about security and sustainability of the codebase. If
> we
> > > had
> > > > > > talked
> > > > > > > > > > > about this last year when that article came out I'd
> have
> > > > agreed
> > > > > > it is
> > > > > > > > > > > too early. Interestingly that link seems to get updated
> > > and I
> > > > > > tried
> > > > > > > > > > > [1] and found more recent data (not sure how recent).
> > > Anyway
> > > > it
> > > > > > > > > > > suggests Java 8 is still the top dog but we see good
> growth
> > > > on
> > > > > > 11. In
> > > > > > > > > > > my $dayjob this aligns to what I'm seeing too.
> Customers
> > > > didn't
> > > > > > seem
> > > > > > > > > > > to care about Java 11 until later half last year and
> now
> > > > > > suddenly it
> > > > > > > > > > > is all over the place.
> > > > > > > > > > >
> > > > > > > > > > > I think once we put out a NiFi 2.0 release we'd see
> rapid
> > > > > > decrease in
> > > > > > > > > > > work on the 1.x line just being blunt. We did this many
> > > years
> > > > > ago
> > > > > > > > > > > with 0.x to 1.x and we stood behind 0.x for a while
> (maybe
> > > a
> > > > > > year or
> > > > > > > > > > > so) but it was purely bug fix/security related bits. We
> > > would
> > > > > > need to
> > > > > > > > > > > do something similar. But feature work would almost
> > > certainly
> > > > > go
> > > > > > to
> > > > > > > > > > > the 2.x line. Maybe there are other workable models but
> my
> > > > > > instinct
> > > > > > > > > > > suggests this is likely to follow a similar path.
> > > > > > > > > > >
> > > > > > > > > > > ...anyway I agree it isn't that easy of a call to dump
> Java
> > > > 8.
> > > > > We
> > > > > > > > > > > need to make the call in both the interests of the user
> > > base
> > > > > and
> > > > > > the
> > > > > > > > > > > contributor base of the community.
> > > > > > > > > > >
> > > > > > > > > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > > Joe
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
> > > joe.witt@gmail.com
> > > > > > <mailto:
> > > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > > > Russ
> > > > > > > > > > >
> > > > > > > > > > > Yeah the flow registry is a key part of it. But also
> now
> > > you
> > > > > can
> > > > > > > > > > > download the flow definition in JSON (upload i think is
> > > there
> > > > > now
> > > > > > > > > > > too). Templates offered a series of challenges such as
> we
> > > > store
> > > > > > them
> > > > > > > > > > > in the flow definition which has made flows massive in
> an
> > > > > > unintended
> > > > > > > > > > > way which isn't fun for cluster behavior.
> > > > > > > > > > >
> > > > > > > > > > > We have a couple cases where we headed down a
> particular
> > > > > concept
> > > > > > and
> > > > > > > > > > > came up with better approaches later. We need to
> reconcile
> > > > > these
> > > > > > with
> > > > > > > > > > > the benefit of hindsight, and while being careful to be
> not
> > > > > > overly
> > > > > > > > > > > disruptive to existing users, to reduce the
> > > > > codebase/maintenance
> > > > > > > > > > > burden and allow continued evolution of the project.
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > > > > > > > russ@windofkeltia.com
> > > > > > > > > > > <ma...@windofkeltia.com>>
> > > > > > > > > > > wrote:
> > > > > > > > > > > Joe,
> > > > > > > > > > >
> > > > > > > > > > > I apologize for the off-topic intrusion, but what
> replaces
> > > > > > templates?
> > > > > > > > > > > The Registry? Templates rocked and we have used them
> since
> > > > > 0.5.x.
> > > > > > > > > > >
> > > > > > > > > > > Russ
> > > > > > > > > > >
> > > > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > > > > > > > David,
> > > > > > > > > > >
> > > > > > > > > > > I think this is a highly reasonable approach and such a
> > > focus
> > > > > > will
> > > > > > > > > > > greatly help make a 2.0 release far more approachable
> to
> > > > knock
> > > > > > out.
> > > > > > > > > > > Not only that but tech debt reduction would help make
> work
> > > > > > towards
> > > > > > > > > > > major features we'd think about in a 'major release'
> sense
> > > > more
> > > > > > > > > > > approachable.
> > > > > > > > > > >
> > > > > > > > > > > We should remove all deprecated things (as well as
> verify
> > > we
> > > > > > have the
> > > > > > > > > > > right list). We should remove/consider removal of
> > > deprecated
> > > > > > > > > > > concepts
> > > > > > > > > > > like templates. We should consider whether we can
> resolve
> > > the
> > > > > > > > > > > various
> > > > > > > > > > > ways we've handled what are now parameters down to one
> > > clean
> > > > > > > > > > > approach.
> > > > > > > > > > > We should remove options in the nifi.properties which
> turn
> > > > out
> > > > > to
> > > > > > > > > > > never be used quite right (if there are). There is
> quite a
> > > > bit
> > > > > we
> > > > > > > > > > > can
> > > > > > > > > > > do purely in the name of tech debt reduction.
> > > > > > > > > > >
> > > > > > > > > > > Lots to consider here but I think this is the right
> > > > discussion.
> > > > > > > > > > >
> > > > > > > > > > > Than ks
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> > > > bbende@gmail.com
> > > > > > > > <mailto:
> > > > > > > > > > > bbende@gmail.com>>
> > > > > > > > > > > wrote:
> > > > > > > > > > > I'm a +1 for this... Not sure if this falls under
> "Removing
> > > > > > > > > > > Deprecated
> > > > > > > > > > > Components", but I think we should also look at
> anything
> > > that
> > > > > has
> > > > > > > > > > > been
> > > > > > > > > > > marked as deprecated throughout the code base as a
> > > candidate
> > > > > for
> > > > > > > > > > > removal. There are quite a few classes, methods,
> > > properties,
> > > > > etc
> > > > > > > > > > > that
> > > > > > > > > > > have been waiting for a chance to be removed.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > > > > > > > > <exceptionfactory@apache.org<mailto:
> > > > > exceptionfactory@apache.org
> > > > > > >>
> > > > > > > > > wrote:
> > > > > > > > > > > Team,
> > > > > > > > > > >
> > > > > > > > > > > With all of the excellent work that many have
> contributed
> > > to
> > > > > NiFi
> > > > > > > > > > > over the
> > > > > > > > > > > years, the code base has also accumulated some amount
> of
> > > > > > technical
> > > > > > > > > > > debt. A
> > > > > > > > > > > handful of components have been marked as deprecated,
> and
> > > > some
> > > > > > > > > > > components
> > > > > > > > > > > remain in the code base to support integration with old
> > > > > versions
> > > > > > > > > > > of various
> > > > > > > > > > > products. Following the principles of semantic
> versioning,
> > > > > > > > > > > introducing a
> > > > > > > > > > > major release would provide the opportunity to remove
> these
> > > > > > > > > > > deprecated and
> > > > > > > > > > > unsupported components.
> > > > > > > > > > >
> > > > > > > > > > > Rather than focusing the next major release on new
> > > features,
> > > > > what
> > > > > > > > > > > do you
> > > > > > > > > > > think about focusing on technical debt removal? This
> > > approach
> > > > > > > > > > > would not
> > > > > > > > > > > make for the most interesting release, but it provides
> the
> > > > > > > > > > > opportunity to
> > > > > > > > > > > clean up elements that involve breaking changes.
> > > > > > > > > > >
> > > > > > > > > > > Focusing on technical debt, at least three primary
> goals
> > > come
> > > > > to
> > > > > > > > > > > mind for
> > > > > > > > > > > the next major release:
> > > > > > > > > > >
> > > > > > > > > > > 1. Removal of deprecated and unmaintained components
> > > > > > > > > > > 2. Require Java 11 as the minimum supported version
> > > > > > > > > > > 3. Transition internal date and time handling to JSR
> 310
> > > > > > java.time
> > > > > > > > > > > components
> > > > > > > > > > >
> > > > > > > > > > > *Removing Deprecated Components*
> > > > > > > > > > >
> > > > > > > > > > > Removing support for older and deprecated components
> > > > provides a
> > > > > > > > > > > great
> > > > > > > > > > > opportunity to improve the overall security posture
> when it
> > > > > comes
> > > > > > > > > > > to
> > > > > > > > > > > maintaining dependencies. The OWASP dependency plugin
> > > report
> > > > > > > > > > > currently
> > > > > > > > > > > generates 50 MB of HTML for questionable dependencies,
> many
> > > > of
> > > > > > > > > > > which are
> > > > > > > > > > > related to old versions of various libraries.
> > > > > > > > > > >
> > > > > > > > > > > As a starting point, here are a handful of components
> and
> > > > > > > > > > > extension modules
> > > > > > > > > > > that could be targeted for removal in a major version:
> > > > > > > > > > >
> > > > > > > > > > > - PostHTTP and GetHTTP
> > > > > > > > > > > - ListenLumberjack and the entire
> nifi-lumberjack-bundle
> > > > > > > > > > > - ListenBeats and the entire nifi-beats-bundle
> > > > > > > > > > > - Elasticsearch 5 components
> > > > > > > > > > > - Hive 1 and 2 components
> > > > > > > > > > >
> > > > > > > > > > > *Requiring Java 11*
> > > > > > > > > > >
> > > > > > > > > > > Java 8 is now over seven years old, and NiFi has
> supported
> > > > > > general
> > > > > > > > > > > compatibility with Java 11 for several years. NiFi
> 1.14.0
> > > > > > > > > > > incorporated
> > > > > > > > > > > internal improvements specifically related to TLS 1.3,
> > > which
> > > > > > > > > > > allowed
> > > > > > > > > > > closing out the long-running Java 11 compatibility epic
> > > > > > NIFI-5174.
> > > > > > > > > > > Making
> > > > > > > > > > > Java 11 the minimum required version provides the
> > > opportunity
> > > > > to
> > > > > > > > > > > address
> > > > > > > > > > > any lingering edge cases and put NiFi in a better
> position
> > > to
> > > > > > > > > > > support
> > > > > > > > > > > current Java versions.
> > > > > > > > > > >
> > > > > > > > > > > *JSR 310 for Date and Time Handling*
> > > > > > > > > > >
> > > > > > > > > > > Without making the scope too broad, transitioning
> internal
> > > > date
> > > > > > > > > > > and time
> > > > > > > > > > > handling to use DateTimeFormatter instead of
> > > SimpleDateFormat
> > > > > > > > > > > would provide
> > > > > > > > > > > a number of advantages. The Java Time components
> provide
> > > much
> > > > > > > > > > > better
> > > > > > > > > > > clarity when it comes to handling localized date and
> time
> > > > > > > > > > > representations,
> > > > > > > > > > > and also avoid the inherent confusion of java.sql.Date
> > > > > extending
> > > > > > > > > > > java.util.Date. Many internal components, specifically
> > > > > > > > > > > Record-oriented
> > > > > > > > > > > processors and services, rely on date parsing, leading
> to
> > > > > > > > > > > confusion and
> > > > > > > > > > > various workarounds. The pattern formats of
> > > SimpleDateFormat
> > > > > and
> > > > > > > > > > > DateTimeFormatter are very similar, but there are a few
> > > > subtle
> > > > > > > > > > > differences.
> > > > > > > > > > > Making this transition would provide a much better
> > > foundation
> > > > > > > > > > > going forward.
> > > > > > > > > > > *Conclusion*
> > > > > > > > > > >
> > > > > > > > > > > Thanks for giving this proposal some consideration.
> Many of
> > > > you
> > > > > > > > > > > have been
> > > > > > > > > > > developing NiFi for years and I look forward to your
> > > > feedback.
> > > > > I
> > > > > > > > > > > would be
> > > > > > > > > > > glad to put together a more formalized recommendation
> on
> > > > > > > > > > > Confluence and
> > > > > > > > > > > write up Jira epics if this general approach sounds
> > > agreeable
> > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > community.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > David Handermann
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Otto Fowler <ot...@gmail.com>.
I find myself wishing I had a list of all the jiras / issues that have
been put off for a 2.0 release because they required some change or another
:(
From: Joe Witt <jo...@gmail.com> <jo...@gmail.com>
Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
Date: July 27, 2021 at 12:30:35
To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
A few thoughts:
1. I would love to see deprecation notices show up in the UI in
various ways to help motivate users to move off things to more
supportable things. That is not a prerequisite for anything happening
however. Just a good feature/nice thing to do for users when someone
is able to tackle it.
2. The decision to deprecate something and to further remove it need
not mean there is a superior solution available. If that thing itself
isn't getting the love/attention it needs to be
maintained/supported/healthy going forward that alone is enough to
remove it. That might well be the case with PostHTTP [1] and for
comparison you can see how much effort has gone into InvokeHTTP [2].
3. When discussing a 2.0 release each thing we add as a 'must do' the
further away from reality such a release will become. We'll have to
get very specific about 'musts' vs 'wants'.
[1]
https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
[2]
https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
On Tue, Jul 27, 2021 at 8:47 AM David Handermann
<ex...@apache.org> wrote:
>
> Thanks Mark, providing a template or comparison statistics with Java
> versions and component configuration details would be very helpful. If it
> is possible to run tests using a public API or deployable service, that
> would also help confirm potential differences.
>
> Showing a deprecation notice in the UI could be helpful, perhaps as a
> configurable option. NIFI-8650 describes a general Flow Analysis
> capability, and it seems like that might be one possible way to surface
> deprecation warnings. For something more specific to component
deprecation,
> it seems important to find a balance between making it obvious and making
> it something that ends up getting ignored.
>
> Regards,
> David Handermann
>
> On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <ma...@gmail.com> wrote:
>
> > I'll start a new thread for PostHTTP when I get a template and/or
detailed
> > stats.
> >
> > I know the deprecation is noted in the documentation. That's a
necessary
> > and minimum level of notification. I was suggesting it be more obvious
in
> > the UI. I think it would be beneficial to somehow be aware of the
> > deprecation status simply by looking at the flow (perhaps even on the
> > summary pages too), and without having to open the documentation for
every
> > processor to confirm whether or not a component is marked as
deprecated.
> >
> > Thanks,
> > Mark
> >
> >
> > On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
> > exceptionfactory@apache.org> wrote:
> >
> > > Mark,
> > >
> > > Thanks for the feedback. It may be better to start a separate thread
on
> > > PostHTTP, but can you provide an example flow demonstrating the
> > performance
> > > differences between PostHTTP and InvokeHTTP?
> > >
> > > PostHTTP uses the Apache HttpComponents library, whereas InvokeHTTP
uses
> > > OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for
OkHttp,
> > > so it would be important to test with the most recent version. It is
also
> > > worth noting that test classes for PostHTTP are now integration tests
as
> > > opposed to unit tests, which means they are not executed as part of
the
> > > automated builds.
> > >
> > > The NiFi documentation includes the contents of the DeprecationNotice
for
> > > PostHTTP:
> > >
> > >
> > >
> >
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <ma...@gmail.com>
wrote:
> > >
> > > > I'm strongly in favor of reducing tech debt, and moving
deliberately
> > to a
> > > > 2.0 release. I have a concern with just one processor that is
currently
> > > > marked as deprecated: PostHTTP. (I have not evaluated specifically
any
> > > > other deprecated components; I cannot say if there are or are not
> > similar
> > > > issues elsewhere.) I understand the rationale for deprecating this
> > > > processor in that it eliminates a processor whose functionality is
> > > > available elsewhere, specifically in the more flexible InvokeHTTP.
> > > However,
> > > > in my experience and testing, PostHTTP performs transfers with far
> > > greater
> > > > throughput than InvokeHTTP. I would not be in favor of removing
> > PostHTTP
> > > > unless/until InvokeHTTP is refactored to increase its throughput
> > > > capability.
> > > >
> > > > Has anyone else continued to use PostHTTP over InvokeHTTP for
similar
> > > > reasons? Or, is there a performance-related configuration for
> > InvokeHTTP
> > > I
> > > > may have missed?
> > > >
> > > > Also, in order to help facilitate a smooth transition to 2.0 from a
> > user
> > > > perspective, would it be advisable to add some sort of visual
indicator
> > > in
> > > > the UI for components that are currently annotated as @Deprecated?
This
> > > > might help users proactively modify their flow prior to a release
that
> > > > would otherwise break it.
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <bb...@gmail.com>
wrote:
> > > >
> > > > > With the merging of MiNiFi and registry into the NiFi repo, we've
> > > > > actually gone more towards a mono-repo where everything is
released
> > > > > together. Eventually it would still be nice to have a smaller
base
> > > > > distribution containing just the framework and standard NARs, but
it
> > > > > is hard to tackle that until we provide a way for users to easily
> > > > > obtain the NARs in some other way.
> > > > >
> > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <
edward.armes@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Given the major version number shift and the spliting up of
> > > processors
> > > > > into
> > > > > > multiple NAR's. I'd like to suggest that we start individually
> > > > versioning
> > > > > > individual NARs/ bundles.
> > > > > >
> > > > > > I can see this bringing a large number of benifits including
making
> > > > Nifi
> > > > > > more deployable with things RPM, but also potentially allowing
> > those
> > > > that
> > > > > > have to deploy managed Nifi instances easier to upgrade.
> > > > > >
> > > > > > Edward
> > > > > >
> > > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, <ot...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > The issue with updating the aws sdk is if it breaks any one
of
> > the
> > > > > > > processors.
> > > > > > > the Web Gateway API invoke processor for example is not using
a
> > > high
> > > > > level
> > > > > > > purpose build client and may break.
> > > > > > >
> > > > > > > If we change the aws version, we need to coordinate in such a
way
> > > > that
> > > > > they
> > > > > > > all
> > > > > > > can come along reasonably.
> > > > > > > IE: what happens if 1 or 2 break but the rest or OK?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: David Handermann <ex...@apache.org>
> > > > > > > <ex...@apache.org>
> > > > > > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > dev@nifi.apache.org>
> > > > > > > Date: July 26, 2021 at 09:33:42
> > > > > > > To: dev@nifi.apache.org <de...@nifi.apache.org> <
> > dev@nifi.apache.org
> > > >
> > > > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > > > > >
> > > > > > > Chris,
> > > > > > >
> > > > > > > Thanks for the reply and recommendations. It seems like some
of
> > the
> > > > > work to
> > > > > > > reorganize the module structure could be done outside of a
major
> > > > > release,
> > > > > > > but it would be great to target any breaking changes for 2.0.
> > > > Perhaps a
> > > > > > > separate feature proposal on module restructuring, with the
goal
> > of
> > > > > > > supporting optimized builds, would be a helpful way to move
that
> > > part
> > > > > of
> > > > > > > the discussion forward.
> > > > > > >
> > > > > > > Regarding updating AWS SDK to version 2, it seems like that
might
> > > be
> > > > > > > possible now. I haven't taken a close look at the referencing
> > > > > components,
> > > > > > > so I'm not sure about the level of effort involved. Minor
NiFi
> > > > version
> > > > > > > updates have incorporated major new versions of dependencies.
For
> > > > > example,
> > > > > > > NiFi 1.14 included an upgrade from Spring Framework 4 to 5.
On
> > the
> > > > one
> > > > > > > hand, including the AWS SDK update as part of a major release
> > seems
> > > > > > > helpful, but unless there are changes that break existing
> > component
> > > > > > > properties, upgrading the AWS SDK could be worked
independently.
> > > > > Others may
> > > > > > > have more insight into particular usage of that library.
> > > > > > >
> > > > > > > Regards,
> > > > > > > David Handermann
> > > > > > >
> > > > > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > > > > > > <ch...@naimuri.com.invalid> wrote:
> > > > > > >
> > > > > > > > Might be worth considering refactoring the build as part of
> > this
> > > > work
> > > > > > > too,
> > > > > > > > e.g. only building the bits of the repo affected by a
commit,
> > > etc.
> > > > -
> > > > > > > > discussed briefly in previous threads but don't think any
> > changes
> > > > > made
> > > > > > > yet.
> > > > > > > > If NARs/components are likely to be split up and refactored
> > then
> > > > such
> > > > > > > work
> > > > > > > > around the build probably makes sense to consider.
> > > > > > > >
> > > > > > > > I've a couple of PRs open that include updates to
Elasticsearch
> > > > > versions
> > > > > > > > already, although I stopped at 7.10.2 (after which Elastic
> > > changed
> > > > > > > licence)
> > > > > > > > in case there were licence concerns. But more work can be
done
> > to
> > > > > tidy up
> > > > > > > > the processors, absolutely.
> > > > > > > >
> > > > > > > > AWS libraries to v2 would seem a sensible move and a
refactor
> > of
> > > > > those
> > > > > > > > processors as well.
> > > > > > > >
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris Sampson
> > > > > > > >
> > > > > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > > > > > > exceptionfactory@apache.org>
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for pointing out the standard NAR bundles, Mark.
There
> > > > are a
> > > > > > > > number
> > > > > > > > > of components in the standard NAR bundles with particular
> > > > > dependencies
> > > > > > > > that
> > > > > > > > > would make more sense in separate NARs. Reorganizing the
> > > standard
> > > > > NAR
> > > > > > > to
> > > > > > > > > components with limited dependencies and wide
applicability
> > > would
> > > > > > > > > definitely help with future maintenance.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > David Handermann
> > > > > > > > >
> > > > > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> > > > markap14@hotmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > There’s also some code that exists in order to maintain
> > > > backward
> > > > > > > > > > compatibility in the repositories. I would very much
like
> > the
> > > > > > > > > repositories
> > > > > > > > > > to contain no unnecessary code. And swap file format
> > supports
> > > > > really
> > > > > > > > old
> > > > > > > > > > formats. And the old impls of the repositories
themselves,
> > > like
> > > > > > > > > > PersistentProvRepo instead of WriteAheadProv Repo, etc.
> > Lots
> > > of
> > > > > stuff
> > > > > > > > > there
> > > > > > > > > > that can be removed. And some methods in ProcessSession
> > that
> > > > are
> > > > > > > never
> > > > > > > > > used
> > > > > > > > > > by any processor in the codebase but exists in the
public
> > API
> > > > so
> > > > > > > can’t
> > > > > > > > be
> > > > > > > > > > removed till 2.0.
> > > > > > > > > >
> > > > > > > > > > I think his is also a great time to clean up the
“standard
> > > > nar.”
> > > > > At
> > > > > > > > this
> > > > > > > > > > point, it’s something like 70 MB. And many of the
> > components
> > > > > there
> > > > > > > are
> > > > > > > > > not
> > > > > > > > > > really “standard” - things like connecting to FTP &
SFTP
> > > > > servers, XML
> > > > > > > > > > processing, Jolt transform, etc. could potentially be
moved
> > > > into
> > > > > > > other
> > > > > > > > > > nars. The
nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
> > is
> > > > > 6.9 MB
> > > > > > > is
> > > > > > > > > not
> > > > > > > > > > necessary for stateless or minifi java. Lots of things
> > > probably
> > > > > to
> > > > > > > > > > reconsider within the standard nar.
> > > > > > > > > >
> > > > > > > > > > I definitely think this is a reasonable approach, to
allow
> > > for
> > > > a
> > > > > 2.0
> > > > > > > > that
> > > > > > > > > > is not a huge feature release but allows the project to
be
> > > > > simpler
> > > > > > > and
> > > > > > > > > more
> > > > > > > > > > nimble.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > -Mark
> > > > > > > > > >
> > > > > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > > > > mikerthomsen@gmail.com
> > > > > > > > > <mailto:
> > > > > > > > > > mikerthomsen@gmail.com>> wrote:
> > > > > > > > > >
> > > > > > > > > > Russell,
> > > > > > > > > >
> > > > > > > > > > AFAICT from looking at Elastic's repos, the low level
REST
> > > > > client is
> > > > > > > > > > still fine.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > > > > > > > >
> > > > > > > > > > Our Elasticsearch support is spread over two NARs at
> > present.
> > > > One
> > > > > > > uses
> > > > > > > > > > OkHttp and the other uses that low level Elastic REST
> > client.
> > > > > > > > > > Therefore, I think we're fine on licensing for the
moment.
> > > > > > > > > >
> > > > > > > > > > Mike
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > > > > > > russ@windofkeltia.com
> > > > > > > > > > <ma...@windofkeltia.com>> wrote:
> > > > > > > > > >
> > > > > > > > > > Bringing up Elastic also reminds me that the Elastic
> > > framework
> > > > > has
> > > > > > > just
> > > > > > > > > > recently transitioned out of Open Source, so to
acknowledge
> > > > that,
> > > > > > > maybe
> > > > > > > > > > some effort toward OpenSearch--I say this not
understanding
> > > > > exactly
> > > > > > > how
> > > > > > > > > > this sort of thing is considered in a large-scale,
> > > world-class
> > > > > > > software
> > > > > > > > > > project like Apache NiFi. (I'm not a contributor, just
a
> > > > grateful
> > > > > > > > > > consumer.)
> > > > > > > > > >
> > > > > > > > > > Russ
> > > > > > > > > >
> > > > > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > > > > > > > > Along with the itemized list for ancient components we
> > should
> > > > > look at
> > > > > > > > > > updating versions of drivers, SDKs, etc. for external
> > systems
> > > > > such as
> > > > > > > > > > Elasticsearch, Cassandra, etc. There may be breaking
> > changes
> > > > but
> > > > > 2.0
> > > > > > > > > > is probably the right time to get things up to date to
make
> > > > them
> > > > > more
> > > > > > > > > > useful to more people.
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > > > > thenatog@gmail.com
> > > > > > > > > <mailto:
> > > > > > > > > > thenatog@gmail.com>> wrote:
> > > > > > > > > > I'm a +1 for removing pretty much all of this stuff.
There
> > > are
> > > > > > > security
> > > > > > > > > > implications to keeping old dependencies around, so the
> > more
> > > > old
> > > > > code
> > > > > > > > we
> > > > > > > > > > can remove the better. I agree that eventually we need
to
> > > move
> > > > to
> > > > > > > > > > supporting only Java 11+, and as our next release will
> > > probably
> > > > > be
> > > > > > > > about
> > > > > > > > > 4
> > > > > > > > > > - 6 months from now that doesn't seem too soon. We
could
> > > > > potentially
> > > > > > > > > break
> > > > > > > > > > this in two and remove the deprecated processors and
leave
> > > 1.x
> > > > on
> > > > > > > Java
> > > > > > > > 8,
> > > > > > > > > > and finally start on 2.x which would support only Java
11.
> > > I'm
> > > > > unsure
> > > > > > > > of
> > > > > > > > > > what implications changing the date and time handling
would
> > > > have
> > > > > -
> > > > > > > for
> > > > > > > > > > running systems that use long term historical logs,
> > > unexpected
> > > > > > > impacts
> > > > > > > > to
> > > > > > > > > > time logging could be a problem.
> > > > > > > > > >
> > > > > > > > > > As Joe says I think feature work will have to be
dedicated
> > to
> > > > > 2.x and
> > > > > > > > we
> > > > > > > > > > could support 1.x for security fixes for some period of
> > time.
> > > > 2.x
> > > > > > > seems
> > > > > > > > > > like a gargantuan task but it's probably time to get
> > started.
> > > > Not
> > > > > > > sure
> > > > > > > > > how
> > > > > > > > > > we handle all open PRs and the transition between 1.x
and
> > > 2.x.
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> > > joe.witt@gmail.com
> > > > > > > <mailto:
> > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > >
> > > > > > > > > > Jon
> > > > > > > > > >
> > > > > > > > > > You're right we have to be careful and you're right
there
> > are
> > > > > still
> > > > > > > > > > significant Java 8 users out there. But we also have to
be
> > > > > careful
> > > > > > > > > > about security and sustainability of the codebase. If
we
> > had
> > > > > talked
> > > > > > > > > > about this last year when that article came out I'd
have
> > > agreed
> > > > > it is
> > > > > > > > > > too early. Interestingly that link seems to get updated
> > and I
> > > > > tried
> > > > > > > > > > [1] and found more recent data (not sure how recent).
> > Anyway
> > > it
> > > > > > > > > > suggests Java 8 is still the top dog but we see good
growth
> > > on
> > > > > 11. In
> > > > > > > > > > my $dayjob this aligns to what I'm seeing too.
Customers
> > > didn't
> > > > > seem
> > > > > > > > > > to care about Java 11 until later half last year and
now
> > > > > suddenly it
> > > > > > > > > > is all over the place.
> > > > > > > > > >
> > > > > > > > > > I think once we put out a NiFi 2.0 release we'd see
rapid
> > > > > decrease in
> > > > > > > > > > work on the 1.x line just being blunt. We did this many
> > years
> > > > ago
> > > > > > > > > > with 0.x to 1.x and we stood behind 0.x for a while
(maybe
> > a
> > > > > year or
> > > > > > > > > > so) but it was purely bug fix/security related bits. We
> > would
> > > > > need to
> > > > > > > > > > do something similar. But feature work would almost
> > certainly
> > > > go
> > > > > to
> > > > > > > > > > the 2.x line. Maybe there are other workable models but
my
> > > > > instinct
> > > > > > > > > > suggests this is likely to follow a similar path.
> > > > > > > > > >
> > > > > > > > > > ...anyway I agree it isn't that easy of a call to dump
Java
> > > 8.
> > > > We
> > > > > > > > > > need to make the call in both the interests of the user
> > base
> > > > and
> > > > > the
> > > > > > > > > > contributor base of the community.
> > > > > > > > > >
> > > > > > > > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Joe
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
> > joe.witt@gmail.com
> > > > > <mailto:
> > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > > Russ
> > > > > > > > > >
> > > > > > > > > > Yeah the flow registry is a key part of it. But also
now
> > you
> > > > can
> > > > > > > > > > download the flow definition in JSON (upload i think is
> > there
> > > > now
> > > > > > > > > > too). Templates offered a series of challenges such as
we
> > > store
> > > > > them
> > > > > > > > > > in the flow definition which has made flows massive in
an
> > > > > unintended
> > > > > > > > > > way which isn't fun for cluster behavior.
> > > > > > > > > >
> > > > > > > > > > We have a couple cases where we headed down a
particular
> > > > concept
> > > > > and
> > > > > > > > > > came up with better approaches later. We need to
reconcile
> > > > these
> > > > > with
> > > > > > > > > > the benefit of hindsight, and while being careful to be
not
> > > > > overly
> > > > > > > > > > disruptive to existing users, to reduce the
> > > > codebase/maintenance
> > > > > > > > > > burden and allow continued evolution of the project.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > > > > > > russ@windofkeltia.com
> > > > > > > > > > <ma...@windofkeltia.com>>
> > > > > > > > > > wrote:
> > > > > > > > > > Joe,
> > > > > > > > > >
> > > > > > > > > > I apologize for the off-topic intrusion, but what
replaces
> > > > > templates?
> > > > > > > > > > The Registry? Templates rocked and we have used them
since
> > > > 0.5.x.
> > > > > > > > > >
> > > > > > > > > > Russ
> > > > > > > > > >
> > > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > > > > > > David,
> > > > > > > > > >
> > > > > > > > > > I think this is a highly reasonable approach and such a
> > focus
> > > > > will
> > > > > > > > > > greatly help make a 2.0 release far more approachable
to
> > > knock
> > > > > out.
> > > > > > > > > > Not only that but tech debt reduction would help make
work
> > > > > towards
> > > > > > > > > > major features we'd think about in a 'major release'
sense
> > > more
> > > > > > > > > > approachable.
> > > > > > > > > >
> > > > > > > > > > We should remove all deprecated things (as well as
verify
> > we
> > > > > have the
> > > > > > > > > > right list). We should remove/consider removal of
> > deprecated
> > > > > > > > > > concepts
> > > > > > > > > > like templates. We should consider whether we can
resolve
> > the
> > > > > > > > > > various
> > > > > > > > > > ways we've handled what are now parameters down to one
> > clean
> > > > > > > > > > approach.
> > > > > > > > > > We should remove options in the nifi.properties which
turn
> > > out
> > > > to
> > > > > > > > > > never be used quite right (if there are). There is
quite a
> > > bit
> > > > we
> > > > > > > > > > can
> > > > > > > > > > do purely in the name of tech debt reduction.
> > > > > > > > > >
> > > > > > > > > > Lots to consider here but I think this is the right
> > > discussion.
> > > > > > > > > >
> > > > > > > > > > Than ks
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> > > bbende@gmail.com
> > > > > > > <mailto:
> > > > > > > > > > bbende@gmail.com>>
> > > > > > > > > > wrote:
> > > > > > > > > > I'm a +1 for this... Not sure if this falls under
"Removing
> > > > > > > > > > Deprecated
> > > > > > > > > > Components", but I think we should also look at
anything
> > that
> > > > has
> > > > > > > > > > been
> > > > > > > > > > marked as deprecated throughout the code base as a
> > candidate
> > > > for
> > > > > > > > > > removal. There are quite a few classes, methods,
> > properties,
> > > > etc
> > > > > > > > > > that
> > > > > > > > > > have been waiting for a chance to be removed.
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > > > > > > > <exceptionfactory@apache.org<mailto:
> > > > exceptionfactory@apache.org
> > > > > >>
> > > > > > > > wrote:
> > > > > > > > > > Team,
> > > > > > > > > >
> > > > > > > > > > With all of the excellent work that many have
contributed
> > to
> > > > NiFi
> > > > > > > > > > over the
> > > > > > > > > > years, the code base has also accumulated some amount
of
> > > > > technical
> > > > > > > > > > debt. A
> > > > > > > > > > handful of components have been marked as deprecated,
and
> > > some
> > > > > > > > > > components
> > > > > > > > > > remain in the code base to support integration with old
> > > > versions
> > > > > > > > > > of various
> > > > > > > > > > products. Following the principles of semantic
versioning,
> > > > > > > > > > introducing a
> > > > > > > > > > major release would provide the opportunity to remove
these
> > > > > > > > > > deprecated and
> > > > > > > > > > unsupported components.
> > > > > > > > > >
> > > > > > > > > > Rather than focusing the next major release on new
> > features,
> > > > what
> > > > > > > > > > do you
> > > > > > > > > > think about focusing on technical debt removal? This
> > approach
> > > > > > > > > > would not
> > > > > > > > > > make for the most interesting release, but it provides
the
> > > > > > > > > > opportunity to
> > > > > > > > > > clean up elements that involve breaking changes.
> > > > > > > > > >
> > > > > > > > > > Focusing on technical debt, at least three primary
goals
> > come
> > > > to
> > > > > > > > > > mind for
> > > > > > > > > > the next major release:
> > > > > > > > > >
> > > > > > > > > > 1. Removal of deprecated and unmaintained components
> > > > > > > > > > 2. Require Java 11 as the minimum supported version
> > > > > > > > > > 3. Transition internal date and time handling to JSR
310
> > > > > java.time
> > > > > > > > > > components
> > > > > > > > > >
> > > > > > > > > > *Removing Deprecated Components*
> > > > > > > > > >
> > > > > > > > > > Removing support for older and deprecated components
> > > provides a
> > > > > > > > > > great
> > > > > > > > > > opportunity to improve the overall security posture
when it
> > > > comes
> > > > > > > > > > to
> > > > > > > > > > maintaining dependencies. The OWASP dependency plugin
> > report
> > > > > > > > > > currently
> > > > > > > > > > generates 50 MB of HTML for questionable dependencies,
many
> > > of
> > > > > > > > > > which are
> > > > > > > > > > related to old versions of various libraries.
> > > > > > > > > >
> > > > > > > > > > As a starting point, here are a handful of components
and
> > > > > > > > > > extension modules
> > > > > > > > > > that could be targeted for removal in a major version:
> > > > > > > > > >
> > > > > > > > > > - PostHTTP and GetHTTP
> > > > > > > > > > - ListenLumberjack and the entire
nifi-lumberjack-bundle
> > > > > > > > > > - ListenBeats and the entire nifi-beats-bundle
> > > > > > > > > > - Elasticsearch 5 components
> > > > > > > > > > - Hive 1 and 2 components
> > > > > > > > > >
> > > > > > > > > > *Requiring Java 11*
> > > > > > > > > >
> > > > > > > > > > Java 8 is now over seven years old, and NiFi has
supported
> > > > > general
> > > > > > > > > > compatibility with Java 11 for several years. NiFi
1.14.0
> > > > > > > > > > incorporated
> > > > > > > > > > internal improvements specifically related to TLS 1.3,
> > which
> > > > > > > > > > allowed
> > > > > > > > > > closing out the long-running Java 11 compatibility epic
> > > > > NIFI-5174.
> > > > > > > > > > Making
> > > > > > > > > > Java 11 the minimum required version provides the
> > opportunity
> > > > to
> > > > > > > > > > address
> > > > > > > > > > any lingering edge cases and put NiFi in a better
position
> > to
> > > > > > > > > > support
> > > > > > > > > > current Java versions.
> > > > > > > > > >
> > > > > > > > > > *JSR 310 for Date and Time Handling*
> > > > > > > > > >
> > > > > > > > > > Without making the scope too broad, transitioning
internal
> > > date
> > > > > > > > > > and time
> > > > > > > > > > handling to use DateTimeFormatter instead of
> > SimpleDateFormat
> > > > > > > > > > would provide
> > > > > > > > > > a number of advantages. The Java Time components
provide
> > much
> > > > > > > > > > better
> > > > > > > > > > clarity when it comes to handling localized date and
time
> > > > > > > > > > representations,
> > > > > > > > > > and also avoid the inherent confusion of java.sql.Date
> > > > extending
> > > > > > > > > > java.util.Date. Many internal components, specifically
> > > > > > > > > > Record-oriented
> > > > > > > > > > processors and services, rely on date parsing, leading
to
> > > > > > > > > > confusion and
> > > > > > > > > > various workarounds. The pattern formats of
> > SimpleDateFormat
> > > > and
> > > > > > > > > > DateTimeFormatter are very similar, but there are a few
> > > subtle
> > > > > > > > > > differences.
> > > > > > > > > > Making this transition would provide a much better
> > foundation
> > > > > > > > > > going forward.
> > > > > > > > > > *Conclusion*
> > > > > > > > > >
> > > > > > > > > > Thanks for giving this proposal some consideration.
Many of
> > > you
> > > > > > > > > > have been
> > > > > > > > > > developing NiFi for years and I look forward to your
> > > feedback.
> > > > I
> > > > > > > > > > would be
> > > > > > > > > > glad to put together a more formalized recommendation
on
> > > > > > > > > > Confluence and
> > > > > > > > > > write up Jira epics if this general approach sounds
> > agreeable
> > > > to
> > > > > > > > > > the
> > > > > > > > > > community.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > David Handermann
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Joe Witt <jo...@gmail.com>.
A few thoughts:
1. I would love to see deprecation notices show up in the UI in
various ways to help motivate users to move off things to more
supportable things. That is not a prerequisite for anything happening
however. Just a good feature/nice thing to do for users when someone
is able to tackle it.
2. The decision to deprecate something and to further remove it need
not mean there is a superior solution available. If that thing itself
isn't getting the love/attention it needs to be
maintained/supported/healthy going forward that alone is enough to
remove it. That might well be the case with PostHTTP [1] and for
comparison you can see how much effort has gone into InvokeHTTP [2].
3. When discussing a 2.0 release each thing we add as a 'must do' the
further away from reality such a release will become. We'll have to
get very specific about 'musts' vs 'wants'.
[1] https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
[2] https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
On Tue, Jul 27, 2021 at 8:47 AM David Handermann
<ex...@apache.org> wrote:
>
> Thanks Mark, providing a template or comparison statistics with Java
> versions and component configuration details would be very helpful. If it
> is possible to run tests using a public API or deployable service, that
> would also help confirm potential differences.
>
> Showing a deprecation notice in the UI could be helpful, perhaps as a
> configurable option. NIFI-8650 describes a general Flow Analysis
> capability, and it seems like that might be one possible way to surface
> deprecation warnings. For something more specific to component deprecation,
> it seems important to find a balance between making it obvious and making
> it something that ends up getting ignored.
>
> Regards,
> David Handermann
>
> On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <ma...@gmail.com> wrote:
>
> > I'll start a new thread for PostHTTP when I get a template and/or detailed
> > stats.
> >
> > I know the deprecation is noted in the documentation. That's a necessary
> > and minimum level of notification. I was suggesting it be more obvious in
> > the UI. I think it would be beneficial to somehow be aware of the
> > deprecation status simply by looking at the flow (perhaps even on the
> > summary pages too), and without having to open the documentation for every
> > processor to confirm whether or not a component is marked as deprecated.
> >
> > Thanks,
> > Mark
> >
> >
> > On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
> > exceptionfactory@apache.org> wrote:
> >
> > > Mark,
> > >
> > > Thanks for the feedback. It may be better to start a separate thread on
> > > PostHTTP, but can you provide an example flow demonstrating the
> > performance
> > > differences between PostHTTP and InvokeHTTP?
> > >
> > > PostHTTP uses the Apache HttpComponents library, whereas InvokeHTTP uses
> > > OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for OkHttp,
> > > so it would be important to test with the most recent version. It is also
> > > worth noting that test classes for PostHTTP are now integration tests as
> > > opposed to unit tests, which means they are not executed as part of the
> > > automated builds.
> > >
> > > The NiFi documentation includes the contents of the DeprecationNotice for
> > > PostHTTP:
> > >
> > >
> > >
> > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <ma...@gmail.com> wrote:
> > >
> > > > I'm strongly in favor of reducing tech debt, and moving deliberately
> > to a
> > > > 2.0 release. I have a concern with just one processor that is currently
> > > > marked as deprecated: PostHTTP. (I have not evaluated specifically any
> > > > other deprecated components; I cannot say if there are or are not
> > similar
> > > > issues elsewhere.) I understand the rationale for deprecating this
> > > > processor in that it eliminates a processor whose functionality is
> > > > available elsewhere, specifically in the more flexible InvokeHTTP.
> > > However,
> > > > in my experience and testing, PostHTTP performs transfers with far
> > > greater
> > > > throughput than InvokeHTTP. I would not be in favor of removing
> > PostHTTP
> > > > unless/until InvokeHTTP is refactored to increase its throughput
> > > > capability.
> > > >
> > > > Has anyone else continued to use PostHTTP over InvokeHTTP for similar
> > > > reasons? Or, is there a performance-related configuration for
> > InvokeHTTP
> > > I
> > > > may have missed?
> > > >
> > > > Also, in order to help facilitate a smooth transition to 2.0 from a
> > user
> > > > perspective, would it be advisable to add some sort of visual indicator
> > > in
> > > > the UI for components that are currently annotated as @Deprecated? This
> > > > might help users proactively modify their flow prior to a release that
> > > > would otherwise break it.
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <bb...@gmail.com> wrote:
> > > >
> > > > > With the merging of MiNiFi and registry into the NiFi repo, we've
> > > > > actually gone more towards a mono-repo where everything is released
> > > > > together. Eventually it would still be nice to have a smaller base
> > > > > distribution containing just the framework and standard NARs, but it
> > > > > is hard to tackle that until we provide a way for users to easily
> > > > > obtain the NARs in some other way.
> > > > >
> > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <edward.armes@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Given the major version number shift and the spliting up of
> > > processors
> > > > > into
> > > > > > multiple NAR's. I'd like to suggest that we start individually
> > > > versioning
> > > > > > individual NARs/ bundles.
> > > > > >
> > > > > > I can see this bringing a large number of benifits including making
> > > > Nifi
> > > > > > more deployable with things RPM, but also potentially allowing
> > those
> > > > that
> > > > > > have to deploy managed Nifi instances easier to upgrade.
> > > > > >
> > > > > > Edward
> > > > > >
> > > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, <ot...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > The issue with updating the aws sdk is if it breaks any one of
> > the
> > > > > > > processors.
> > > > > > > the Web Gateway API invoke processor for example is not using a
> > > high
> > > > > level
> > > > > > > purpose build client and may break.
> > > > > > >
> > > > > > > If we change the aws version, we need to coordinate in such a way
> > > > that
> > > > > they
> > > > > > > all
> > > > > > > can come along reasonably.
> > > > > > > IE: what happens if 1 or 2 break but the rest or OK?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: David Handermann <ex...@apache.org>
> > > > > > > <ex...@apache.org>
> > > > > > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > > dev@nifi.apache.org>
> > > > > > > Date: July 26, 2021 at 09:33:42
> > > > > > > To: dev@nifi.apache.org <de...@nifi.apache.org> <
> > dev@nifi.apache.org
> > > >
> > > > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > > > > >
> > > > > > > Chris,
> > > > > > >
> > > > > > > Thanks for the reply and recommendations. It seems like some of
> > the
> > > > > work to
> > > > > > > reorganize the module structure could be done outside of a major
> > > > > release,
> > > > > > > but it would be great to target any breaking changes for 2.0.
> > > > Perhaps a
> > > > > > > separate feature proposal on module restructuring, with the goal
> > of
> > > > > > > supporting optimized builds, would be a helpful way to move that
> > > part
> > > > > of
> > > > > > > the discussion forward.
> > > > > > >
> > > > > > > Regarding updating AWS SDK to version 2, it seems like that might
> > > be
> > > > > > > possible now. I haven't taken a close look at the referencing
> > > > > components,
> > > > > > > so I'm not sure about the level of effort involved. Minor NiFi
> > > > version
> > > > > > > updates have incorporated major new versions of dependencies. For
> > > > > example,
> > > > > > > NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On
> > the
> > > > one
> > > > > > > hand, including the AWS SDK update as part of a major release
> > seems
> > > > > > > helpful, but unless there are changes that break existing
> > component
> > > > > > > properties, upgrading the AWS SDK could be worked independently.
> > > > > Others may
> > > > > > > have more insight into particular usage of that library.
> > > > > > >
> > > > > > > Regards,
> > > > > > > David Handermann
> > > > > > >
> > > > > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > > > > > > <ch...@naimuri.com.invalid> wrote:
> > > > > > >
> > > > > > > > Might be worth considering refactoring the build as part of
> > this
> > > > work
> > > > > > > too,
> > > > > > > > e.g. only building the bits of the repo affected by a commit,
> > > etc.
> > > > -
> > > > > > > > discussed briefly in previous threads but don't think any
> > changes
> > > > > made
> > > > > > > yet.
> > > > > > > > If NARs/components are likely to be split up and refactored
> > then
> > > > such
> > > > > > > work
> > > > > > > > around the build probably makes sense to consider.
> > > > > > > >
> > > > > > > > I've a couple of PRs open that include updates to Elasticsearch
> > > > > versions
> > > > > > > > already, although I stopped at 7.10.2 (after which Elastic
> > > changed
> > > > > > > licence)
> > > > > > > > in case there were licence concerns. But more work can be done
> > to
> > > > > tidy up
> > > > > > > > the processors, absolutely.
> > > > > > > >
> > > > > > > > AWS libraries to v2 would seem a sensible move and a refactor
> > of
> > > > > those
> > > > > > > > processors as well.
> > > > > > > >
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris Sampson
> > > > > > > >
> > > > > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > > > > > > exceptionfactory@apache.org>
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for pointing out the standard NAR bundles, Mark. There
> > > > are a
> > > > > > > > number
> > > > > > > > > of components in the standard NAR bundles with particular
> > > > > dependencies
> > > > > > > > that
> > > > > > > > > would make more sense in separate NARs. Reorganizing the
> > > standard
> > > > > NAR
> > > > > > > to
> > > > > > > > > components with limited dependencies and wide applicability
> > > would
> > > > > > > > > definitely help with future maintenance.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > David Handermann
> > > > > > > > >
> > > > > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> > > > markap14@hotmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > There’s also some code that exists in order to maintain
> > > > backward
> > > > > > > > > > compatibility in the repositories. I would very much like
> > the
> > > > > > > > > repositories
> > > > > > > > > > to contain no unnecessary code. And swap file format
> > supports
> > > > > really
> > > > > > > > old
> > > > > > > > > > formats. And the old impls of the repositories themselves,
> > > like
> > > > > > > > > > PersistentProvRepo instead of WriteAheadProv Repo, etc.
> > Lots
> > > of
> > > > > stuff
> > > > > > > > > there
> > > > > > > > > > that can be removed. And some methods in ProcessSession
> > that
> > > > are
> > > > > > > never
> > > > > > > > > used
> > > > > > > > > > by any processor in the codebase but exists in the public
> > API
> > > > so
> > > > > > > can’t
> > > > > > > > be
> > > > > > > > > > removed till 2.0.
> > > > > > > > > >
> > > > > > > > > > I think his is also a great time to clean up the “standard
> > > > nar.”
> > > > > At
> > > > > > > > this
> > > > > > > > > > point, it’s something like 70 MB. And many of the
> > components
> > > > > there
> > > > > > > are
> > > > > > > > > not
> > > > > > > > > > really “standard” - things like connecting to FTP & SFTP
> > > > > servers, XML
> > > > > > > > > > processing, Jolt transform, etc. could potentially be moved
> > > > into
> > > > > > > other
> > > > > > > > > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
> > is
> > > > > 6.9 MB
> > > > > > > is
> > > > > > > > > not
> > > > > > > > > > necessary for stateless or minifi java. Lots of things
> > > probably
> > > > > to
> > > > > > > > > > reconsider within the standard nar.
> > > > > > > > > >
> > > > > > > > > > I definitely think this is a reasonable approach, to allow
> > > for
> > > > a
> > > > > 2.0
> > > > > > > > that
> > > > > > > > > > is not a huge feature release but allows the project to be
> > > > > simpler
> > > > > > > and
> > > > > > > > > more
> > > > > > > > > > nimble.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > -Mark
> > > > > > > > > >
> > > > > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > > > > mikerthomsen@gmail.com
> > > > > > > > > <mailto:
> > > > > > > > > > mikerthomsen@gmail.com>> wrote:
> > > > > > > > > >
> > > > > > > > > > Russell,
> > > > > > > > > >
> > > > > > > > > > AFAICT from looking at Elastic's repos, the low level REST
> > > > > client is
> > > > > > > > > > still fine.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> > https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > > > > > > > >
> > > > > > > > > > Our Elasticsearch support is spread over two NARs at
> > present.
> > > > One
> > > > > > > uses
> > > > > > > > > > OkHttp and the other uses that low level Elastic REST
> > client.
> > > > > > > > > > Therefore, I think we're fine on licensing for the moment.
> > > > > > > > > >
> > > > > > > > > > Mike
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > > > > > > russ@windofkeltia.com
> > > > > > > > > > <ma...@windofkeltia.com>> wrote:
> > > > > > > > > >
> > > > > > > > > > Bringing up Elastic also reminds me that the Elastic
> > > framework
> > > > > has
> > > > > > > just
> > > > > > > > > > recently transitioned out of Open Source, so to acknowledge
> > > > that,
> > > > > > > maybe
> > > > > > > > > > some effort toward OpenSearch--I say this not understanding
> > > > > exactly
> > > > > > > how
> > > > > > > > > > this sort of thing is considered in a large-scale,
> > > world-class
> > > > > > > software
> > > > > > > > > > project like Apache NiFi. (I'm not a contributor, just a
> > > > grateful
> > > > > > > > > > consumer.)
> > > > > > > > > >
> > > > > > > > > > Russ
> > > > > > > > > >
> > > > > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > > > > > > > > Along with the itemized list for ancient components we
> > should
> > > > > look at
> > > > > > > > > > updating versions of drivers, SDKs, etc. for external
> > systems
> > > > > such as
> > > > > > > > > > Elasticsearch, Cassandra, etc. There may be breaking
> > changes
> > > > but
> > > > > 2.0
> > > > > > > > > > is probably the right time to get things up to date to make
> > > > them
> > > > > more
> > > > > > > > > > useful to more people.
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > > > > thenatog@gmail.com
> > > > > > > > > <mailto:
> > > > > > > > > > thenatog@gmail.com>> wrote:
> > > > > > > > > > I'm a +1 for removing pretty much all of this stuff. There
> > > are
> > > > > > > security
> > > > > > > > > > implications to keeping old dependencies around, so the
> > more
> > > > old
> > > > > code
> > > > > > > > we
> > > > > > > > > > can remove the better. I agree that eventually we need to
> > > move
> > > > to
> > > > > > > > > > supporting only Java 11+, and as our next release will
> > > probably
> > > > > be
> > > > > > > > about
> > > > > > > > > 4
> > > > > > > > > > - 6 months from now that doesn't seem too soon. We could
> > > > > potentially
> > > > > > > > > break
> > > > > > > > > > this in two and remove the deprecated processors and leave
> > > 1.x
> > > > on
> > > > > > > Java
> > > > > > > > 8,
> > > > > > > > > > and finally start on 2.x which would support only Java 11.
> > > I'm
> > > > > unsure
> > > > > > > > of
> > > > > > > > > > what implications changing the date and time handling would
> > > > have
> > > > > -
> > > > > > > for
> > > > > > > > > > running systems that use long term historical logs,
> > > unexpected
> > > > > > > impacts
> > > > > > > > to
> > > > > > > > > > time logging could be a problem.
> > > > > > > > > >
> > > > > > > > > > As Joe says I think feature work will have to be dedicated
> > to
> > > > > 2.x and
> > > > > > > > we
> > > > > > > > > > could support 1.x for security fixes for some period of
> > time.
> > > > 2.x
> > > > > > > seems
> > > > > > > > > > like a gargantuan task but it's probably time to get
> > started.
> > > > Not
> > > > > > > sure
> > > > > > > > > how
> > > > > > > > > > we handle all open PRs and the transition between 1.x and
> > > 2.x.
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> > > joe.witt@gmail.com
> > > > > > > <mailto:
> > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > >
> > > > > > > > > > Jon
> > > > > > > > > >
> > > > > > > > > > You're right we have to be careful and you're right there
> > are
> > > > > still
> > > > > > > > > > significant Java 8 users out there. But we also have to be
> > > > > careful
> > > > > > > > > > about security and sustainability of the codebase. If we
> > had
> > > > > talked
> > > > > > > > > > about this last year when that article came out I'd have
> > > agreed
> > > > > it is
> > > > > > > > > > too early. Interestingly that link seems to get updated
> > and I
> > > > > tried
> > > > > > > > > > [1] and found more recent data (not sure how recent).
> > Anyway
> > > it
> > > > > > > > > > suggests Java 8 is still the top dog but we see good growth
> > > on
> > > > > 11. In
> > > > > > > > > > my $dayjob this aligns to what I'm seeing too. Customers
> > > didn't
> > > > > seem
> > > > > > > > > > to care about Java 11 until later half last year and now
> > > > > suddenly it
> > > > > > > > > > is all over the place.
> > > > > > > > > >
> > > > > > > > > > I think once we put out a NiFi 2.0 release we'd see rapid
> > > > > decrease in
> > > > > > > > > > work on the 1.x line just being blunt. We did this many
> > years
> > > > ago
> > > > > > > > > > with 0.x to 1.x and we stood behind 0.x for a while (maybe
> > a
> > > > > year or
> > > > > > > > > > so) but it was purely bug fix/security related bits. We
> > would
> > > > > need to
> > > > > > > > > > do something similar. But feature work would almost
> > certainly
> > > > go
> > > > > to
> > > > > > > > > > the 2.x line. Maybe there are other workable models but my
> > > > > instinct
> > > > > > > > > > suggests this is likely to follow a similar path.
> > > > > > > > > >
> > > > > > > > > > ...anyway I agree it isn't that easy of a call to dump Java
> > > 8.
> > > > We
> > > > > > > > > > need to make the call in both the interests of the user
> > base
> > > > and
> > > > > the
> > > > > > > > > > contributor base of the community.
> > > > > > > > > >
> > > > > > > > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Joe
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
> > joe.witt@gmail.com
> > > > > <mailto:
> > > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > > Russ
> > > > > > > > > >
> > > > > > > > > > Yeah the flow registry is a key part of it. But also now
> > you
> > > > can
> > > > > > > > > > download the flow definition in JSON (upload i think is
> > there
> > > > now
> > > > > > > > > > too). Templates offered a series of challenges such as we
> > > store
> > > > > them
> > > > > > > > > > in the flow definition which has made flows massive in an
> > > > > unintended
> > > > > > > > > > way which isn't fun for cluster behavior.
> > > > > > > > > >
> > > > > > > > > > We have a couple cases where we headed down a particular
> > > > concept
> > > > > and
> > > > > > > > > > came up with better approaches later. We need to reconcile
> > > > these
> > > > > with
> > > > > > > > > > the benefit of hindsight, and while being careful to be not
> > > > > overly
> > > > > > > > > > disruptive to existing users, to reduce the
> > > > codebase/maintenance
> > > > > > > > > > burden and allow continued evolution of the project.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > > > > > > russ@windofkeltia.com
> > > > > > > > > > <ma...@windofkeltia.com>>
> > > > > > > > > > wrote:
> > > > > > > > > > Joe,
> > > > > > > > > >
> > > > > > > > > > I apologize for the off-topic intrusion, but what replaces
> > > > > templates?
> > > > > > > > > > The Registry? Templates rocked and we have used them since
> > > > 0.5.x.
> > > > > > > > > >
> > > > > > > > > > Russ
> > > > > > > > > >
> > > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > > > > > > David,
> > > > > > > > > >
> > > > > > > > > > I think this is a highly reasonable approach and such a
> > focus
> > > > > will
> > > > > > > > > > greatly help make a 2.0 release far more approachable to
> > > knock
> > > > > out.
> > > > > > > > > > Not only that but tech debt reduction would help make work
> > > > > towards
> > > > > > > > > > major features we'd think about in a 'major release' sense
> > > more
> > > > > > > > > > approachable.
> > > > > > > > > >
> > > > > > > > > > We should remove all deprecated things (as well as verify
> > we
> > > > > have the
> > > > > > > > > > right list). We should remove/consider removal of
> > deprecated
> > > > > > > > > > concepts
> > > > > > > > > > like templates. We should consider whether we can resolve
> > the
> > > > > > > > > > various
> > > > > > > > > > ways we've handled what are now parameters down to one
> > clean
> > > > > > > > > > approach.
> > > > > > > > > > We should remove options in the nifi.properties which turn
> > > out
> > > > to
> > > > > > > > > > never be used quite right (if there are). There is quite a
> > > bit
> > > > we
> > > > > > > > > > can
> > > > > > > > > > do purely in the name of tech debt reduction.
> > > > > > > > > >
> > > > > > > > > > Lots to consider here but I think this is the right
> > > discussion.
> > > > > > > > > >
> > > > > > > > > > Than ks
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> > > bbende@gmail.com
> > > > > > > <mailto:
> > > > > > > > > > bbende@gmail.com>>
> > > > > > > > > > wrote:
> > > > > > > > > > I'm a +1 for this... Not sure if this falls under "Removing
> > > > > > > > > > Deprecated
> > > > > > > > > > Components", but I think we should also look at anything
> > that
> > > > has
> > > > > > > > > > been
> > > > > > > > > > marked as deprecated throughout the code base as a
> > candidate
> > > > for
> > > > > > > > > > removal. There are quite a few classes, methods,
> > properties,
> > > > etc
> > > > > > > > > > that
> > > > > > > > > > have been waiting for a chance to be removed.
> > > > > > > > > >
> > > > > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > > > > > > > <exceptionfactory@apache.org<mailto:
> > > > exceptionfactory@apache.org
> > > > > >>
> > > > > > > > wrote:
> > > > > > > > > > Team,
> > > > > > > > > >
> > > > > > > > > > With all of the excellent work that many have contributed
> > to
> > > > NiFi
> > > > > > > > > > over the
> > > > > > > > > > years, the code base has also accumulated some amount of
> > > > > technical
> > > > > > > > > > debt. A
> > > > > > > > > > handful of components have been marked as deprecated, and
> > > some
> > > > > > > > > > components
> > > > > > > > > > remain in the code base to support integration with old
> > > > versions
> > > > > > > > > > of various
> > > > > > > > > > products. Following the principles of semantic versioning,
> > > > > > > > > > introducing a
> > > > > > > > > > major release would provide the opportunity to remove these
> > > > > > > > > > deprecated and
> > > > > > > > > > unsupported components.
> > > > > > > > > >
> > > > > > > > > > Rather than focusing the next major release on new
> > features,
> > > > what
> > > > > > > > > > do you
> > > > > > > > > > think about focusing on technical debt removal? This
> > approach
> > > > > > > > > > would not
> > > > > > > > > > make for the most interesting release, but it provides the
> > > > > > > > > > opportunity to
> > > > > > > > > > clean up elements that involve breaking changes.
> > > > > > > > > >
> > > > > > > > > > Focusing on technical debt, at least three primary goals
> > come
> > > > to
> > > > > > > > > > mind for
> > > > > > > > > > the next major release:
> > > > > > > > > >
> > > > > > > > > > 1. Removal of deprecated and unmaintained components
> > > > > > > > > > 2. Require Java 11 as the minimum supported version
> > > > > > > > > > 3. Transition internal date and time handling to JSR 310
> > > > > java.time
> > > > > > > > > > components
> > > > > > > > > >
> > > > > > > > > > *Removing Deprecated Components*
> > > > > > > > > >
> > > > > > > > > > Removing support for older and deprecated components
> > > provides a
> > > > > > > > > > great
> > > > > > > > > > opportunity to improve the overall security posture when it
> > > > comes
> > > > > > > > > > to
> > > > > > > > > > maintaining dependencies. The OWASP dependency plugin
> > report
> > > > > > > > > > currently
> > > > > > > > > > generates 50 MB of HTML for questionable dependencies, many
> > > of
> > > > > > > > > > which are
> > > > > > > > > > related to old versions of various libraries.
> > > > > > > > > >
> > > > > > > > > > As a starting point, here are a handful of components and
> > > > > > > > > > extension modules
> > > > > > > > > > that could be targeted for removal in a major version:
> > > > > > > > > >
> > > > > > > > > > - PostHTTP and GetHTTP
> > > > > > > > > > - ListenLumberjack and the entire nifi-lumberjack-bundle
> > > > > > > > > > - ListenBeats and the entire nifi-beats-bundle
> > > > > > > > > > - Elasticsearch 5 components
> > > > > > > > > > - Hive 1 and 2 components
> > > > > > > > > >
> > > > > > > > > > *Requiring Java 11*
> > > > > > > > > >
> > > > > > > > > > Java 8 is now over seven years old, and NiFi has supported
> > > > > general
> > > > > > > > > > compatibility with Java 11 for several years. NiFi 1.14.0
> > > > > > > > > > incorporated
> > > > > > > > > > internal improvements specifically related to TLS 1.3,
> > which
> > > > > > > > > > allowed
> > > > > > > > > > closing out the long-running Java 11 compatibility epic
> > > > > NIFI-5174.
> > > > > > > > > > Making
> > > > > > > > > > Java 11 the minimum required version provides the
> > opportunity
> > > > to
> > > > > > > > > > address
> > > > > > > > > > any lingering edge cases and put NiFi in a better position
> > to
> > > > > > > > > > support
> > > > > > > > > > current Java versions.
> > > > > > > > > >
> > > > > > > > > > *JSR 310 for Date and Time Handling*
> > > > > > > > > >
> > > > > > > > > > Without making the scope too broad, transitioning internal
> > > date
> > > > > > > > > > and time
> > > > > > > > > > handling to use DateTimeFormatter instead of
> > SimpleDateFormat
> > > > > > > > > > would provide
> > > > > > > > > > a number of advantages. The Java Time components provide
> > much
> > > > > > > > > > better
> > > > > > > > > > clarity when it comes to handling localized date and time
> > > > > > > > > > representations,
> > > > > > > > > > and also avoid the inherent confusion of java.sql.Date
> > > > extending
> > > > > > > > > > java.util.Date. Many internal components, specifically
> > > > > > > > > > Record-oriented
> > > > > > > > > > processors and services, rely on date parsing, leading to
> > > > > > > > > > confusion and
> > > > > > > > > > various workarounds. The pattern formats of
> > SimpleDateFormat
> > > > and
> > > > > > > > > > DateTimeFormatter are very similar, but there are a few
> > > subtle
> > > > > > > > > > differences.
> > > > > > > > > > Making this transition would provide a much better
> > foundation
> > > > > > > > > > going forward.
> > > > > > > > > > *Conclusion*
> > > > > > > > > >
> > > > > > > > > > Thanks for giving this proposal some consideration. Many of
> > > you
> > > > > > > > > > have been
> > > > > > > > > > developing NiFi for years and I look forward to your
> > > feedback.
> > > > I
> > > > > > > > > > would be
> > > > > > > > > > glad to put together a more formalized recommendation on
> > > > > > > > > > Confluence and
> > > > > > > > > > write up Jira epics if this general approach sounds
> > agreeable
> > > > to
> > > > > > > > > > the
> > > > > > > > > > community.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > David Handermann
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by David Handermann <ex...@apache.org>.
Thanks Mark, providing a template or comparison statistics with Java
versions and component configuration details would be very helpful. If it
is possible to run tests using a public API or deployable service, that
would also help confirm potential differences.
Showing a deprecation notice in the UI could be helpful, perhaps as a
configurable option. NIFI-8650 describes a general Flow Analysis
capability, and it seems like that might be one possible way to surface
deprecation warnings. For something more specific to component deprecation,
it seems important to find a balance between making it obvious and making
it something that ends up getting ignored.
Regards,
David Handermann
On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <ma...@gmail.com> wrote:
> I'll start a new thread for PostHTTP when I get a template and/or detailed
> stats.
>
> I know the deprecation is noted in the documentation. That's a necessary
> and minimum level of notification. I was suggesting it be more obvious in
> the UI. I think it would be beneficial to somehow be aware of the
> deprecation status simply by looking at the flow (perhaps even on the
> summary pages too), and without having to open the documentation for every
> processor to confirm whether or not a component is marked as deprecated.
>
> Thanks,
> Mark
>
>
> On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
> exceptionfactory@apache.org> wrote:
>
> > Mark,
> >
> > Thanks for the feedback. It may be better to start a separate thread on
> > PostHTTP, but can you provide an example flow demonstrating the
> performance
> > differences between PostHTTP and InvokeHTTP?
> >
> > PostHTTP uses the Apache HttpComponents library, whereas InvokeHTTP uses
> > OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for OkHttp,
> > so it would be important to test with the most recent version. It is also
> > worth noting that test classes for PostHTTP are now integration tests as
> > opposed to unit tests, which means they are not executed as part of the
> > automated builds.
> >
> > The NiFi documentation includes the contents of the DeprecationNotice for
> > PostHTTP:
> >
> >
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
> >
> > Regards,
> > David Handermann
> >
> > On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <ma...@gmail.com> wrote:
> >
> > > I'm strongly in favor of reducing tech debt, and moving deliberately
> to a
> > > 2.0 release. I have a concern with just one processor that is currently
> > > marked as deprecated: PostHTTP. (I have not evaluated specifically any
> > > other deprecated components; I cannot say if there are or are not
> similar
> > > issues elsewhere.) I understand the rationale for deprecating this
> > > processor in that it eliminates a processor whose functionality is
> > > available elsewhere, specifically in the more flexible InvokeHTTP.
> > However,
> > > in my experience and testing, PostHTTP performs transfers with far
> > greater
> > > throughput than InvokeHTTP. I would not be in favor of removing
> PostHTTP
> > > unless/until InvokeHTTP is refactored to increase its throughput
> > > capability.
> > >
> > > Has anyone else continued to use PostHTTP over InvokeHTTP for similar
> > > reasons? Or, is there a performance-related configuration for
> InvokeHTTP
> > I
> > > may have missed?
> > >
> > > Also, in order to help facilitate a smooth transition to 2.0 from a
> user
> > > perspective, would it be advisable to add some sort of visual indicator
> > in
> > > the UI for components that are currently annotated as @Deprecated? This
> > > might help users proactively modify their flow prior to a release that
> > > would otherwise break it.
> > >
> > >
> > >
> > >
> > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <bb...@gmail.com> wrote:
> > >
> > > > With the merging of MiNiFi and registry into the NiFi repo, we've
> > > > actually gone more towards a mono-repo where everything is released
> > > > together. Eventually it would still be nice to have a smaller base
> > > > distribution containing just the framework and standard NARs, but it
> > > > is hard to tackle that until we provide a way for users to easily
> > > > obtain the NARs in some other way.
> > > >
> > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <edward.armes@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > Given the major version number shift and the spliting up of
> > processors
> > > > into
> > > > > multiple NAR's. I'd like to suggest that we start individually
> > > versioning
> > > > > individual NARs/ bundles.
> > > > >
> > > > > I can see this bringing a large number of benifits including making
> > > Nifi
> > > > > more deployable with things RPM, but also potentially allowing
> those
> > > that
> > > > > have to deploy managed Nifi instances easier to upgrade.
> > > > >
> > > > > Edward
> > > > >
> > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, <ot...@gmail.com>
> > > wrote:
> > > > >
> > > > > > The issue with updating the aws sdk is if it breaks any one of
> the
> > > > > > processors.
> > > > > > the Web Gateway API invoke processor for example is not using a
> > high
> > > > level
> > > > > > purpose build client and may break.
> > > > > >
> > > > > > If we change the aws version, we need to coordinate in such a way
> > > that
> > > > they
> > > > > > all
> > > > > > can come along reasonably.
> > > > > > IE: what happens if 1 or 2 break but the rest or OK?
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: David Handermann <ex...@apache.org>
> > > > > > <ex...@apache.org>
> > > > > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > > dev@nifi.apache.org>
> > > > > > Date: July 26, 2021 at 09:33:42
> > > > > > To: dev@nifi.apache.org <de...@nifi.apache.org> <
> dev@nifi.apache.org
> > >
> > > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > > > >
> > > > > > Chris,
> > > > > >
> > > > > > Thanks for the reply and recommendations. It seems like some of
> the
> > > > work to
> > > > > > reorganize the module structure could be done outside of a major
> > > > release,
> > > > > > but it would be great to target any breaking changes for 2.0.
> > > Perhaps a
> > > > > > separate feature proposal on module restructuring, with the goal
> of
> > > > > > supporting optimized builds, would be a helpful way to move that
> > part
> > > > of
> > > > > > the discussion forward.
> > > > > >
> > > > > > Regarding updating AWS SDK to version 2, it seems like that might
> > be
> > > > > > possible now. I haven't taken a close look at the referencing
> > > > components,
> > > > > > so I'm not sure about the level of effort involved. Minor NiFi
> > > version
> > > > > > updates have incorporated major new versions of dependencies. For
> > > > example,
> > > > > > NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On
> the
> > > one
> > > > > > hand, including the AWS SDK update as part of a major release
> seems
> > > > > > helpful, but unless there are changes that break existing
> component
> > > > > > properties, upgrading the AWS SDK could be worked independently.
> > > > Others may
> > > > > > have more insight into particular usage of that library.
> > > > > >
> > > > > > Regards,
> > > > > > David Handermann
> > > > > >
> > > > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > > > > > <ch...@naimuri.com.invalid> wrote:
> > > > > >
> > > > > > > Might be worth considering refactoring the build as part of
> this
> > > work
> > > > > > too,
> > > > > > > e.g. only building the bits of the repo affected by a commit,
> > etc.
> > > -
> > > > > > > discussed briefly in previous threads but don't think any
> changes
> > > > made
> > > > > > yet.
> > > > > > > If NARs/components are likely to be split up and refactored
> then
> > > such
> > > > > > work
> > > > > > > around the build probably makes sense to consider.
> > > > > > >
> > > > > > > I've a couple of PRs open that include updates to Elasticsearch
> > > > versions
> > > > > > > already, although I stopped at 7.10.2 (after which Elastic
> > changed
> > > > > > licence)
> > > > > > > in case there were licence concerns. But more work can be done
> to
> > > > tidy up
> > > > > > > the processors, absolutely.
> > > > > > >
> > > > > > > AWS libraries to v2 would seem a sensible move and a refactor
> of
> > > > those
> > > > > > > processors as well.
> > > > > > >
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris Sampson
> > > > > > >
> > > > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > > > > > exceptionfactory@apache.org>
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for pointing out the standard NAR bundles, Mark. There
> > > are a
> > > > > > > number
> > > > > > > > of components in the standard NAR bundles with particular
> > > > dependencies
> > > > > > > that
> > > > > > > > would make more sense in separate NARs. Reorganizing the
> > standard
> > > > NAR
> > > > > > to
> > > > > > > > components with limited dependencies and wide applicability
> > would
> > > > > > > > definitely help with future maintenance.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > David Handermann
> > > > > > > >
> > > > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> > > markap14@hotmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > There’s also some code that exists in order to maintain
> > > backward
> > > > > > > > > compatibility in the repositories. I would very much like
> the
> > > > > > > > repositories
> > > > > > > > > to contain no unnecessary code. And swap file format
> supports
> > > > really
> > > > > > > old
> > > > > > > > > formats. And the old impls of the repositories themselves,
> > like
> > > > > > > > > PersistentProvRepo instead of WriteAheadProv Repo, etc.
> Lots
> > of
> > > > stuff
> > > > > > > > there
> > > > > > > > > that can be removed. And some methods in ProcessSession
> that
> > > are
> > > > > > never
> > > > > > > > used
> > > > > > > > > by any processor in the codebase but exists in the public
> API
> > > so
> > > > > > can’t
> > > > > > > be
> > > > > > > > > removed till 2.0.
> > > > > > > > >
> > > > > > > > > I think his is also a great time to clean up the “standard
> > > nar.”
> > > > At
> > > > > > > this
> > > > > > > > > point, it’s something like 70 MB. And many of the
> components
> > > > there
> > > > > > are
> > > > > > > > not
> > > > > > > > > really “standard” - things like connecting to FTP & SFTP
> > > > servers, XML
> > > > > > > > > processing, Jolt transform, etc. could potentially be moved
> > > into
> > > > > > other
> > > > > > > > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war
> is
> > > > 6.9 MB
> > > > > > is
> > > > > > > > not
> > > > > > > > > necessary for stateless or minifi java. Lots of things
> > probably
> > > > to
> > > > > > > > > reconsider within the standard nar.
> > > > > > > > >
> > > > > > > > > I definitely think this is a reasonable approach, to allow
> > for
> > > a
> > > > 2.0
> > > > > > > that
> > > > > > > > > is not a huge feature release but allows the project to be
> > > > simpler
> > > > > > and
> > > > > > > > more
> > > > > > > > > nimble.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > -Mark
> > > > > > > > >
> > > > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > > > mikerthomsen@gmail.com
> > > > > > > > <mailto:
> > > > > > > > > mikerthomsen@gmail.com>> wrote:
> > > > > > > > >
> > > > > > > > > Russell,
> > > > > > > > >
> > > > > > > > > AFAICT from looking at Elastic's repos, the low level REST
> > > > client is
> > > > > > > > > still fine.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > > > > > > >
> > > > > > > > > Our Elasticsearch support is spread over two NARs at
> present.
> > > One
> > > > > > uses
> > > > > > > > > OkHttp and the other uses that low level Elastic REST
> client.
> > > > > > > > > Therefore, I think we're fine on licensing for the moment.
> > > > > > > > >
> > > > > > > > > Mike
> > > > > > > > >
> > > > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > > > > > russ@windofkeltia.com
> > > > > > > > > <ma...@windofkeltia.com>> wrote:
> > > > > > > > >
> > > > > > > > > Bringing up Elastic also reminds me that the Elastic
> > framework
> > > > has
> > > > > > just
> > > > > > > > > recently transitioned out of Open Source, so to acknowledge
> > > that,
> > > > > > maybe
> > > > > > > > > some effort toward OpenSearch--I say this not understanding
> > > > exactly
> > > > > > how
> > > > > > > > > this sort of thing is considered in a large-scale,
> > world-class
> > > > > > software
> > > > > > > > > project like Apache NiFi. (I'm not a contributor, just a
> > > grateful
> > > > > > > > > consumer.)
> > > > > > > > >
> > > > > > > > > Russ
> > > > > > > > >
> > > > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > > > > > > > Along with the itemized list for ancient components we
> should
> > > > look at
> > > > > > > > > updating versions of drivers, SDKs, etc. for external
> systems
> > > > such as
> > > > > > > > > Elasticsearch, Cassandra, etc. There may be breaking
> changes
> > > but
> > > > 2.0
> > > > > > > > > is probably the right time to get things up to date to make
> > > them
> > > > more
> > > > > > > > > useful to more people.
> > > > > > > > >
> > > > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > > > thenatog@gmail.com
> > > > > > > > <mailto:
> > > > > > > > > thenatog@gmail.com>> wrote:
> > > > > > > > > I'm a +1 for removing pretty much all of this stuff. There
> > are
> > > > > > security
> > > > > > > > > implications to keeping old dependencies around, so the
> more
> > > old
> > > > code
> > > > > > > we
> > > > > > > > > can remove the better. I agree that eventually we need to
> > move
> > > to
> > > > > > > > > supporting only Java 11+, and as our next release will
> > probably
> > > > be
> > > > > > > about
> > > > > > > > 4
> > > > > > > > > - 6 months from now that doesn't seem too soon. We could
> > > > potentially
> > > > > > > > break
> > > > > > > > > this in two and remove the deprecated processors and leave
> > 1.x
> > > on
> > > > > > Java
> > > > > > > 8,
> > > > > > > > > and finally start on 2.x which would support only Java 11.
> > I'm
> > > > unsure
> > > > > > > of
> > > > > > > > > what implications changing the date and time handling would
> > > have
> > > > -
> > > > > > for
> > > > > > > > > running systems that use long term historical logs,
> > unexpected
> > > > > > impacts
> > > > > > > to
> > > > > > > > > time logging could be a problem.
> > > > > > > > >
> > > > > > > > > As Joe says I think feature work will have to be dedicated
> to
> > > > 2.x and
> > > > > > > we
> > > > > > > > > could support 1.x for security fixes for some period of
> time.
> > > 2.x
> > > > > > seems
> > > > > > > > > like a gargantuan task but it's probably time to get
> started.
> > > Not
> > > > > > sure
> > > > > > > > how
> > > > > > > > > we handle all open PRs and the transition between 1.x and
> > 2.x.
> > > > > > > > >
> > > > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> > joe.witt@gmail.com
> > > > > > <mailto:
> > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > >
> > > > > > > > > Jon
> > > > > > > > >
> > > > > > > > > You're right we have to be careful and you're right there
> are
> > > > still
> > > > > > > > > significant Java 8 users out there. But we also have to be
> > > > careful
> > > > > > > > > about security and sustainability of the codebase. If we
> had
> > > > talked
> > > > > > > > > about this last year when that article came out I'd have
> > agreed
> > > > it is
> > > > > > > > > too early. Interestingly that link seems to get updated
> and I
> > > > tried
> > > > > > > > > [1] and found more recent data (not sure how recent).
> Anyway
> > it
> > > > > > > > > suggests Java 8 is still the top dog but we see good growth
> > on
> > > > 11. In
> > > > > > > > > my $dayjob this aligns to what I'm seeing too. Customers
> > didn't
> > > > seem
> > > > > > > > > to care about Java 11 until later half last year and now
> > > > suddenly it
> > > > > > > > > is all over the place.
> > > > > > > > >
> > > > > > > > > I think once we put out a NiFi 2.0 release we'd see rapid
> > > > decrease in
> > > > > > > > > work on the 1.x line just being blunt. We did this many
> years
> > > ago
> > > > > > > > > with 0.x to 1.x and we stood behind 0.x for a while (maybe
> a
> > > > year or
> > > > > > > > > so) but it was purely bug fix/security related bits. We
> would
> > > > need to
> > > > > > > > > do something similar. But feature work would almost
> certainly
> > > go
> > > > to
> > > > > > > > > the 2.x line. Maybe there are other workable models but my
> > > > instinct
> > > > > > > > > suggests this is likely to follow a similar path.
> > > > > > > > >
> > > > > > > > > ...anyway I agree it isn't that easy of a call to dump Java
> > 8.
> > > We
> > > > > > > > > need to make the call in both the interests of the user
> base
> > > and
> > > > the
> > > > > > > > > contributor base of the community.
> > > > > > > > >
> > > > > > > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Joe
> > > > > > > > >
> > > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <
> joe.witt@gmail.com
> > > > <mailto:
> > > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > > Russ
> > > > > > > > >
> > > > > > > > > Yeah the flow registry is a key part of it. But also now
> you
> > > can
> > > > > > > > > download the flow definition in JSON (upload i think is
> there
> > > now
> > > > > > > > > too). Templates offered a series of challenges such as we
> > store
> > > > them
> > > > > > > > > in the flow definition which has made flows massive in an
> > > > unintended
> > > > > > > > > way which isn't fun for cluster behavior.
> > > > > > > > >
> > > > > > > > > We have a couple cases where we headed down a particular
> > > concept
> > > > and
> > > > > > > > > came up with better approaches later. We need to reconcile
> > > these
> > > > with
> > > > > > > > > the benefit of hindsight, and while being careful to be not
> > > > overly
> > > > > > > > > disruptive to existing users, to reduce the
> > > codebase/maintenance
> > > > > > > > > burden and allow continued evolution of the project.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > > > > > russ@windofkeltia.com
> > > > > > > > > <ma...@windofkeltia.com>>
> > > > > > > > > wrote:
> > > > > > > > > Joe,
> > > > > > > > >
> > > > > > > > > I apologize for the off-topic intrusion, but what replaces
> > > > templates?
> > > > > > > > > The Registry? Templates rocked and we have used them since
> > > 0.5.x.
> > > > > > > > >
> > > > > > > > > Russ
> > > > > > > > >
> > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > > > > > David,
> > > > > > > > >
> > > > > > > > > I think this is a highly reasonable approach and such a
> focus
> > > > will
> > > > > > > > > greatly help make a 2.0 release far more approachable to
> > knock
> > > > out.
> > > > > > > > > Not only that but tech debt reduction would help make work
> > > > towards
> > > > > > > > > major features we'd think about in a 'major release' sense
> > more
> > > > > > > > > approachable.
> > > > > > > > >
> > > > > > > > > We should remove all deprecated things (as well as verify
> we
> > > > have the
> > > > > > > > > right list). We should remove/consider removal of
> deprecated
> > > > > > > > > concepts
> > > > > > > > > like templates. We should consider whether we can resolve
> the
> > > > > > > > > various
> > > > > > > > > ways we've handled what are now parameters down to one
> clean
> > > > > > > > > approach.
> > > > > > > > > We should remove options in the nifi.properties which turn
> > out
> > > to
> > > > > > > > > never be used quite right (if there are). There is quite a
> > bit
> > > we
> > > > > > > > > can
> > > > > > > > > do purely in the name of tech debt reduction.
> > > > > > > > >
> > > > > > > > > Lots to consider here but I think this is the right
> > discussion.
> > > > > > > > >
> > > > > > > > > Than ks
> > > > > > > > >
> > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> > bbende@gmail.com
> > > > > > <mailto:
> > > > > > > > > bbende@gmail.com>>
> > > > > > > > > wrote:
> > > > > > > > > I'm a +1 for this... Not sure if this falls under "Removing
> > > > > > > > > Deprecated
> > > > > > > > > Components", but I think we should also look at anything
> that
> > > has
> > > > > > > > > been
> > > > > > > > > marked as deprecated throughout the code base as a
> candidate
> > > for
> > > > > > > > > removal. There are quite a few classes, methods,
> properties,
> > > etc
> > > > > > > > > that
> > > > > > > > > have been waiting for a chance to be removed.
> > > > > > > > >
> > > > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > > > > > > <exceptionfactory@apache.org<mailto:
> > > exceptionfactory@apache.org
> > > > >>
> > > > > > > wrote:
> > > > > > > > > Team,
> > > > > > > > >
> > > > > > > > > With all of the excellent work that many have contributed
> to
> > > NiFi
> > > > > > > > > over the
> > > > > > > > > years, the code base has also accumulated some amount of
> > > > technical
> > > > > > > > > debt. A
> > > > > > > > > handful of components have been marked as deprecated, and
> > some
> > > > > > > > > components
> > > > > > > > > remain in the code base to support integration with old
> > > versions
> > > > > > > > > of various
> > > > > > > > > products. Following the principles of semantic versioning,
> > > > > > > > > introducing a
> > > > > > > > > major release would provide the opportunity to remove these
> > > > > > > > > deprecated and
> > > > > > > > > unsupported components.
> > > > > > > > >
> > > > > > > > > Rather than focusing the next major release on new
> features,
> > > what
> > > > > > > > > do you
> > > > > > > > > think about focusing on technical debt removal? This
> approach
> > > > > > > > > would not
> > > > > > > > > make for the most interesting release, but it provides the
> > > > > > > > > opportunity to
> > > > > > > > > clean up elements that involve breaking changes.
> > > > > > > > >
> > > > > > > > > Focusing on technical debt, at least three primary goals
> come
> > > to
> > > > > > > > > mind for
> > > > > > > > > the next major release:
> > > > > > > > >
> > > > > > > > > 1. Removal of deprecated and unmaintained components
> > > > > > > > > 2. Require Java 11 as the minimum supported version
> > > > > > > > > 3. Transition internal date and time handling to JSR 310
> > > > java.time
> > > > > > > > > components
> > > > > > > > >
> > > > > > > > > *Removing Deprecated Components*
> > > > > > > > >
> > > > > > > > > Removing support for older and deprecated components
> > provides a
> > > > > > > > > great
> > > > > > > > > opportunity to improve the overall security posture when it
> > > comes
> > > > > > > > > to
> > > > > > > > > maintaining dependencies. The OWASP dependency plugin
> report
> > > > > > > > > currently
> > > > > > > > > generates 50 MB of HTML for questionable dependencies, many
> > of
> > > > > > > > > which are
> > > > > > > > > related to old versions of various libraries.
> > > > > > > > >
> > > > > > > > > As a starting point, here are a handful of components and
> > > > > > > > > extension modules
> > > > > > > > > that could be targeted for removal in a major version:
> > > > > > > > >
> > > > > > > > > - PostHTTP and GetHTTP
> > > > > > > > > - ListenLumberjack and the entire nifi-lumberjack-bundle
> > > > > > > > > - ListenBeats and the entire nifi-beats-bundle
> > > > > > > > > - Elasticsearch 5 components
> > > > > > > > > - Hive 1 and 2 components
> > > > > > > > >
> > > > > > > > > *Requiring Java 11*
> > > > > > > > >
> > > > > > > > > Java 8 is now over seven years old, and NiFi has supported
> > > > general
> > > > > > > > > compatibility with Java 11 for several years. NiFi 1.14.0
> > > > > > > > > incorporated
> > > > > > > > > internal improvements specifically related to TLS 1.3,
> which
> > > > > > > > > allowed
> > > > > > > > > closing out the long-running Java 11 compatibility epic
> > > > NIFI-5174.
> > > > > > > > > Making
> > > > > > > > > Java 11 the minimum required version provides the
> opportunity
> > > to
> > > > > > > > > address
> > > > > > > > > any lingering edge cases and put NiFi in a better position
> to
> > > > > > > > > support
> > > > > > > > > current Java versions.
> > > > > > > > >
> > > > > > > > > *JSR 310 for Date and Time Handling*
> > > > > > > > >
> > > > > > > > > Without making the scope too broad, transitioning internal
> > date
> > > > > > > > > and time
> > > > > > > > > handling to use DateTimeFormatter instead of
> SimpleDateFormat
> > > > > > > > > would provide
> > > > > > > > > a number of advantages. The Java Time components provide
> much
> > > > > > > > > better
> > > > > > > > > clarity when it comes to handling localized date and time
> > > > > > > > > representations,
> > > > > > > > > and also avoid the inherent confusion of java.sql.Date
> > > extending
> > > > > > > > > java.util.Date. Many internal components, specifically
> > > > > > > > > Record-oriented
> > > > > > > > > processors and services, rely on date parsing, leading to
> > > > > > > > > confusion and
> > > > > > > > > various workarounds. The pattern formats of
> SimpleDateFormat
> > > and
> > > > > > > > > DateTimeFormatter are very similar, but there are a few
> > subtle
> > > > > > > > > differences.
> > > > > > > > > Making this transition would provide a much better
> foundation
> > > > > > > > > going forward.
> > > > > > > > > *Conclusion*
> > > > > > > > >
> > > > > > > > > Thanks for giving this proposal some consideration. Many of
> > you
> > > > > > > > > have been
> > > > > > > > > developing NiFi for years and I look forward to your
> > feedback.
> > > I
> > > > > > > > > would be
> > > > > > > > > glad to put together a more formalized recommendation on
> > > > > > > > > Confluence and
> > > > > > > > > write up Jira epics if this general approach sounds
> agreeable
> > > to
> > > > > > > > > the
> > > > > > > > > community.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > David Handermann
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Mark Bean <ma...@gmail.com>.
I'll start a new thread for PostHTTP when I get a template and/or detailed
stats.
I know the deprecation is noted in the documentation. That's a necessary
and minimum level of notification. I was suggesting it be more obvious in
the UI. I think it would be beneficial to somehow be aware of the
deprecation status simply by looking at the flow (perhaps even on the
summary pages too), and without having to open the documentation for every
processor to confirm whether or not a component is marked as deprecated.
Thanks,
Mark
On Tue, Jul 27, 2021 at 11:16 AM David Handermann <
exceptionfactory@apache.org> wrote:
> Mark,
>
> Thanks for the feedback. It may be better to start a separate thread on
> PostHTTP, but can you provide an example flow demonstrating the performance
> differences between PostHTTP and InvokeHTTP?
>
> PostHTTP uses the Apache HttpComponents library, whereas InvokeHTTP uses
> OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for OkHttp,
> so it would be important to test with the most recent version. It is also
> worth noting that test classes for PostHTTP are now integration tests as
> opposed to unit tests, which means they are not executed as part of the
> automated builds.
>
> The NiFi documentation includes the contents of the DeprecationNotice for
> PostHTTP:
>
>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
>
> Regards,
> David Handermann
>
> On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <ma...@gmail.com> wrote:
>
> > I'm strongly in favor of reducing tech debt, and moving deliberately to a
> > 2.0 release. I have a concern with just one processor that is currently
> > marked as deprecated: PostHTTP. (I have not evaluated specifically any
> > other deprecated components; I cannot say if there are or are not similar
> > issues elsewhere.) I understand the rationale for deprecating this
> > processor in that it eliminates a processor whose functionality is
> > available elsewhere, specifically in the more flexible InvokeHTTP.
> However,
> > in my experience and testing, PostHTTP performs transfers with far
> greater
> > throughput than InvokeHTTP. I would not be in favor of removing PostHTTP
> > unless/until InvokeHTTP is refactored to increase its throughput
> > capability.
> >
> > Has anyone else continued to use PostHTTP over InvokeHTTP for similar
> > reasons? Or, is there a performance-related configuration for InvokeHTTP
> I
> > may have missed?
> >
> > Also, in order to help facilitate a smooth transition to 2.0 from a user
> > perspective, would it be advisable to add some sort of visual indicator
> in
> > the UI for components that are currently annotated as @Deprecated? This
> > might help users proactively modify their flow prior to a release that
> > would otherwise break it.
> >
> >
> >
> >
> > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <bb...@gmail.com> wrote:
> >
> > > With the merging of MiNiFi and registry into the NiFi repo, we've
> > > actually gone more towards a mono-repo where everything is released
> > > together. Eventually it would still be nice to have a smaller base
> > > distribution containing just the framework and standard NARs, but it
> > > is hard to tackle that until we provide a way for users to easily
> > > obtain the NARs in some other way.
> > >
> > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <ed...@gmail.com>
> > > wrote:
> > > >
> > > > Given the major version number shift and the spliting up of
> processors
> > > into
> > > > multiple NAR's. I'd like to suggest that we start individually
> > versioning
> > > > individual NARs/ bundles.
> > > >
> > > > I can see this bringing a large number of benifits including making
> > Nifi
> > > > more deployable with things RPM, but also potentially allowing those
> > that
> > > > have to deploy managed Nifi instances easier to upgrade.
> > > >
> > > > Edward
> > > >
> > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, <ot...@gmail.com>
> > wrote:
> > > >
> > > > > The issue with updating the aws sdk is if it breaks any one of the
> > > > > processors.
> > > > > the Web Gateway API invoke processor for example is not using a
> high
> > > level
> > > > > purpose build client and may break.
> > > > >
> > > > > If we change the aws version, we need to coordinate in such a way
> > that
> > > they
> > > > > all
> > > > > can come along reasonably.
> > > > > IE: what happens if 1 or 2 break but the rest or OK?
> > > > >
> > > > >
> > > > >
> > > > > From: David Handermann <ex...@apache.org>
> > > > > <ex...@apache.org>
> > > > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > dev@nifi.apache.org>
> > > > > Date: July 26, 2021 at 09:33:42
> > > > > To: dev@nifi.apache.org <de...@nifi.apache.org> <dev@nifi.apache.org
> >
> > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > > >
> > > > > Chris,
> > > > >
> > > > > Thanks for the reply and recommendations. It seems like some of the
> > > work to
> > > > > reorganize the module structure could be done outside of a major
> > > release,
> > > > > but it would be great to target any breaking changes for 2.0.
> > Perhaps a
> > > > > separate feature proposal on module restructuring, with the goal of
> > > > > supporting optimized builds, would be a helpful way to move that
> part
> > > of
> > > > > the discussion forward.
> > > > >
> > > > > Regarding updating AWS SDK to version 2, it seems like that might
> be
> > > > > possible now. I haven't taken a close look at the referencing
> > > components,
> > > > > so I'm not sure about the level of effort involved. Minor NiFi
> > version
> > > > > updates have incorporated major new versions of dependencies. For
> > > example,
> > > > > NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the
> > one
> > > > > hand, including the AWS SDK update as part of a major release seems
> > > > > helpful, but unless there are changes that break existing component
> > > > > properties, upgrading the AWS SDK could be worked independently.
> > > Others may
> > > > > have more insight into particular usage of that library.
> > > > >
> > > > > Regards,
> > > > > David Handermann
> > > > >
> > > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > > > > <ch...@naimuri.com.invalid> wrote:
> > > > >
> > > > > > Might be worth considering refactoring the build as part of this
> > work
> > > > > too,
> > > > > > e.g. only building the bits of the repo affected by a commit,
> etc.
> > -
> > > > > > discussed briefly in previous threads but don't think any changes
> > > made
> > > > > yet.
> > > > > > If NARs/components are likely to be split up and refactored then
> > such
> > > > > work
> > > > > > around the build probably makes sense to consider.
> > > > > >
> > > > > > I've a couple of PRs open that include updates to Elasticsearch
> > > versions
> > > > > > already, although I stopped at 7.10.2 (after which Elastic
> changed
> > > > > licence)
> > > > > > in case there were licence concerns. But more work can be done to
> > > tidy up
> > > > > > the processors, absolutely.
> > > > > >
> > > > > > AWS libraries to v2 would seem a sensible move and a refactor of
> > > those
> > > > > > processors as well.
> > > > > >
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris Sampson
> > > > > >
> > > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > > > > exceptionfactory@apache.org>
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for pointing out the standard NAR bundles, Mark. There
> > are a
> > > > > > number
> > > > > > > of components in the standard NAR bundles with particular
> > > dependencies
> > > > > > that
> > > > > > > would make more sense in separate NARs. Reorganizing the
> standard
> > > NAR
> > > > > to
> > > > > > > components with limited dependencies and wide applicability
> would
> > > > > > > definitely help with future maintenance.
> > > > > > >
> > > > > > > Regards,
> > > > > > > David Handermann
> > > > > > >
> > > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> > markap14@hotmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > There’s also some code that exists in order to maintain
> > backward
> > > > > > > > compatibility in the repositories. I would very much like the
> > > > > > > repositories
> > > > > > > > to contain no unnecessary code. And swap file format supports
> > > really
> > > > > > old
> > > > > > > > formats. And the old impls of the repositories themselves,
> like
> > > > > > > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots
> of
> > > stuff
> > > > > > > there
> > > > > > > > that can be removed. And some methods in ProcessSession that
> > are
> > > > > never
> > > > > > > used
> > > > > > > > by any processor in the codebase but exists in the public API
> > so
> > > > > can’t
> > > > > > be
> > > > > > > > removed till 2.0.
> > > > > > > >
> > > > > > > > I think his is also a great time to clean up the “standard
> > nar.”
> > > At
> > > > > > this
> > > > > > > > point, it’s something like 70 MB. And many of the components
> > > there
> > > > > are
> > > > > > > not
> > > > > > > > really “standard” - things like connecting to FTP & SFTP
> > > servers, XML
> > > > > > > > processing, Jolt transform, etc. could potentially be moved
> > into
> > > > > other
> > > > > > > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is
> > > 6.9 MB
> > > > > is
> > > > > > > not
> > > > > > > > necessary for stateless or minifi java. Lots of things
> probably
> > > to
> > > > > > > > reconsider within the standard nar.
> > > > > > > >
> > > > > > > > I definitely think this is a reasonable approach, to allow
> for
> > a
> > > 2.0
> > > > > > that
> > > > > > > > is not a huge feature release but allows the project to be
> > > simpler
> > > > > and
> > > > > > > more
> > > > > > > > nimble.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > -Mark
> > > > > > > >
> > > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > > mikerthomsen@gmail.com
> > > > > > > <mailto:
> > > > > > > > mikerthomsen@gmail.com>> wrote:
> > > > > > > >
> > > > > > > > Russell,
> > > > > > > >
> > > > > > > > AFAICT from looking at Elastic's repos, the low level REST
> > > client is
> > > > > > > > still fine.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > >
> >
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > > > > > >
> > > > > > > > Our Elasticsearch support is spread over two NARs at present.
> > One
> > > > > uses
> > > > > > > > OkHttp and the other uses that low level Elastic REST client.
> > > > > > > > Therefore, I think we're fine on licensing for the moment.
> > > > > > > >
> > > > > > > > Mike
> > > > > > > >
> > > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > > > > russ@windofkeltia.com
> > > > > > > > <ma...@windofkeltia.com>> wrote:
> > > > > > > >
> > > > > > > > Bringing up Elastic also reminds me that the Elastic
> framework
> > > has
> > > > > just
> > > > > > > > recently transitioned out of Open Source, so to acknowledge
> > that,
> > > > > maybe
> > > > > > > > some effort toward OpenSearch--I say this not understanding
> > > exactly
> > > > > how
> > > > > > > > this sort of thing is considered in a large-scale,
> world-class
> > > > > software
> > > > > > > > project like Apache NiFi. (I'm not a contributor, just a
> > grateful
> > > > > > > > consumer.)
> > > > > > > >
> > > > > > > > Russ
> > > > > > > >
> > > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > > > > > > Along with the itemized list for ancient components we should
> > > look at
> > > > > > > > updating versions of drivers, SDKs, etc. for external systems
> > > such as
> > > > > > > > Elasticsearch, Cassandra, etc. There may be breaking changes
> > but
> > > 2.0
> > > > > > > > is probably the right time to get things up to date to make
> > them
> > > more
> > > > > > > > useful to more people.
> > > > > > > >
> > > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > > thenatog@gmail.com
> > > > > > > <mailto:
> > > > > > > > thenatog@gmail.com>> wrote:
> > > > > > > > I'm a +1 for removing pretty much all of this stuff. There
> are
> > > > > security
> > > > > > > > implications to keeping old dependencies around, so the more
> > old
> > > code
> > > > > > we
> > > > > > > > can remove the better. I agree that eventually we need to
> move
> > to
> > > > > > > > supporting only Java 11+, and as our next release will
> probably
> > > be
> > > > > > about
> > > > > > > 4
> > > > > > > > - 6 months from now that doesn't seem too soon. We could
> > > potentially
> > > > > > > break
> > > > > > > > this in two and remove the deprecated processors and leave
> 1.x
> > on
> > > > > Java
> > > > > > 8,
> > > > > > > > and finally start on 2.x which would support only Java 11.
> I'm
> > > unsure
> > > > > > of
> > > > > > > > what implications changing the date and time handling would
> > have
> > > -
> > > > > for
> > > > > > > > running systems that use long term historical logs,
> unexpected
> > > > > impacts
> > > > > > to
> > > > > > > > time logging could be a problem.
> > > > > > > >
> > > > > > > > As Joe says I think feature work will have to be dedicated to
> > > 2.x and
> > > > > > we
> > > > > > > > could support 1.x for security fixes for some period of time.
> > 2.x
> > > > > seems
> > > > > > > > like a gargantuan task but it's probably time to get started.
> > Not
> > > > > sure
> > > > > > > how
> > > > > > > > we handle all open PRs and the transition between 1.x and
> 2.x.
> > > > > > > >
> > > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <
> joe.witt@gmail.com
> > > > > <mailto:
> > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > >
> > > > > > > > Jon
> > > > > > > >
> > > > > > > > You're right we have to be careful and you're right there are
> > > still
> > > > > > > > significant Java 8 users out there. But we also have to be
> > > careful
> > > > > > > > about security and sustainability of the codebase. If we had
> > > talked
> > > > > > > > about this last year when that article came out I'd have
> agreed
> > > it is
> > > > > > > > too early. Interestingly that link seems to get updated and I
> > > tried
> > > > > > > > [1] and found more recent data (not sure how recent). Anyway
> it
> > > > > > > > suggests Java 8 is still the top dog but we see good growth
> on
> > > 11. In
> > > > > > > > my $dayjob this aligns to what I'm seeing too. Customers
> didn't
> > > seem
> > > > > > > > to care about Java 11 until later half last year and now
> > > suddenly it
> > > > > > > > is all over the place.
> > > > > > > >
> > > > > > > > I think once we put out a NiFi 2.0 release we'd see rapid
> > > decrease in
> > > > > > > > work on the 1.x line just being blunt. We did this many years
> > ago
> > > > > > > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a
> > > year or
> > > > > > > > so) but it was purely bug fix/security related bits. We would
> > > need to
> > > > > > > > do something similar. But feature work would almost certainly
> > go
> > > to
> > > > > > > > the 2.x line. Maybe there are other workable models but my
> > > instinct
> > > > > > > > suggests this is likely to follow a similar path.
> > > > > > > >
> > > > > > > > ...anyway I agree it isn't that easy of a call to dump Java
> 8.
> > We
> > > > > > > > need to make the call in both the interests of the user base
> > and
> > > the
> > > > > > > > contributor base of the community.
> > > > > > > >
> > > > > > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Joe
> > > > > > > >
> > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <joe.witt@gmail.com
> > > <mailto:
> > > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > > Russ
> > > > > > > >
> > > > > > > > Yeah the flow registry is a key part of it. But also now you
> > can
> > > > > > > > download the flow definition in JSON (upload i think is there
> > now
> > > > > > > > too). Templates offered a series of challenges such as we
> store
> > > them
> > > > > > > > in the flow definition which has made flows massive in an
> > > unintended
> > > > > > > > way which isn't fun for cluster behavior.
> > > > > > > >
> > > > > > > > We have a couple cases where we headed down a particular
> > concept
> > > and
> > > > > > > > came up with better approaches later. We need to reconcile
> > these
> > > with
> > > > > > > > the benefit of hindsight, and while being careful to be not
> > > overly
> > > > > > > > disruptive to existing users, to reduce the
> > codebase/maintenance
> > > > > > > > burden and allow continued evolution of the project.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > > > > russ@windofkeltia.com
> > > > > > > > <ma...@windofkeltia.com>>
> > > > > > > > wrote:
> > > > > > > > Joe,
> > > > > > > >
> > > > > > > > I apologize for the off-topic intrusion, but what replaces
> > > templates?
> > > > > > > > The Registry? Templates rocked and we have used them since
> > 0.5.x.
> > > > > > > >
> > > > > > > > Russ
> > > > > > > >
> > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > > > > David,
> > > > > > > >
> > > > > > > > I think this is a highly reasonable approach and such a focus
> > > will
> > > > > > > > greatly help make a 2.0 release far more approachable to
> knock
> > > out.
> > > > > > > > Not only that but tech debt reduction would help make work
> > > towards
> > > > > > > > major features we'd think about in a 'major release' sense
> more
> > > > > > > > approachable.
> > > > > > > >
> > > > > > > > We should remove all deprecated things (as well as verify we
> > > have the
> > > > > > > > right list). We should remove/consider removal of deprecated
> > > > > > > > concepts
> > > > > > > > like templates. We should consider whether we can resolve the
> > > > > > > > various
> > > > > > > > ways we've handled what are now parameters down to one clean
> > > > > > > > approach.
> > > > > > > > We should remove options in the nifi.properties which turn
> out
> > to
> > > > > > > > never be used quite right (if there are). There is quite a
> bit
> > we
> > > > > > > > can
> > > > > > > > do purely in the name of tech debt reduction.
> > > > > > > >
> > > > > > > > Lots to consider here but I think this is the right
> discussion.
> > > > > > > >
> > > > > > > > Than ks
> > > > > > > >
> > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <
> bbende@gmail.com
> > > > > <mailto:
> > > > > > > > bbende@gmail.com>>
> > > > > > > > wrote:
> > > > > > > > I'm a +1 for this... Not sure if this falls under "Removing
> > > > > > > > Deprecated
> > > > > > > > Components", but I think we should also look at anything that
> > has
> > > > > > > > been
> > > > > > > > marked as deprecated throughout the code base as a candidate
> > for
> > > > > > > > removal. There are quite a few classes, methods, properties,
> > etc
> > > > > > > > that
> > > > > > > > have been waiting for a chance to be removed.
> > > > > > > >
> > > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > > > > > <exceptionfactory@apache.org<mailto:
> > exceptionfactory@apache.org
> > > >>
> > > > > > wrote:
> > > > > > > > Team,
> > > > > > > >
> > > > > > > > With all of the excellent work that many have contributed to
> > NiFi
> > > > > > > > over the
> > > > > > > > years, the code base has also accumulated some amount of
> > > technical
> > > > > > > > debt. A
> > > > > > > > handful of components have been marked as deprecated, and
> some
> > > > > > > > components
> > > > > > > > remain in the code base to support integration with old
> > versions
> > > > > > > > of various
> > > > > > > > products. Following the principles of semantic versioning,
> > > > > > > > introducing a
> > > > > > > > major release would provide the opportunity to remove these
> > > > > > > > deprecated and
> > > > > > > > unsupported components.
> > > > > > > >
> > > > > > > > Rather than focusing the next major release on new features,
> > what
> > > > > > > > do you
> > > > > > > > think about focusing on technical debt removal? This approach
> > > > > > > > would not
> > > > > > > > make for the most interesting release, but it provides the
> > > > > > > > opportunity to
> > > > > > > > clean up elements that involve breaking changes.
> > > > > > > >
> > > > > > > > Focusing on technical debt, at least three primary goals come
> > to
> > > > > > > > mind for
> > > > > > > > the next major release:
> > > > > > > >
> > > > > > > > 1. Removal of deprecated and unmaintained components
> > > > > > > > 2. Require Java 11 as the minimum supported version
> > > > > > > > 3. Transition internal date and time handling to JSR 310
> > > java.time
> > > > > > > > components
> > > > > > > >
> > > > > > > > *Removing Deprecated Components*
> > > > > > > >
> > > > > > > > Removing support for older and deprecated components
> provides a
> > > > > > > > great
> > > > > > > > opportunity to improve the overall security posture when it
> > comes
> > > > > > > > to
> > > > > > > > maintaining dependencies. The OWASP dependency plugin report
> > > > > > > > currently
> > > > > > > > generates 50 MB of HTML for questionable dependencies, many
> of
> > > > > > > > which are
> > > > > > > > related to old versions of various libraries.
> > > > > > > >
> > > > > > > > As a starting point, here are a handful of components and
> > > > > > > > extension modules
> > > > > > > > that could be targeted for removal in a major version:
> > > > > > > >
> > > > > > > > - PostHTTP and GetHTTP
> > > > > > > > - ListenLumberjack and the entire nifi-lumberjack-bundle
> > > > > > > > - ListenBeats and the entire nifi-beats-bundle
> > > > > > > > - Elasticsearch 5 components
> > > > > > > > - Hive 1 and 2 components
> > > > > > > >
> > > > > > > > *Requiring Java 11*
> > > > > > > >
> > > > > > > > Java 8 is now over seven years old, and NiFi has supported
> > > general
> > > > > > > > compatibility with Java 11 for several years. NiFi 1.14.0
> > > > > > > > incorporated
> > > > > > > > internal improvements specifically related to TLS 1.3, which
> > > > > > > > allowed
> > > > > > > > closing out the long-running Java 11 compatibility epic
> > > NIFI-5174.
> > > > > > > > Making
> > > > > > > > Java 11 the minimum required version provides the opportunity
> > to
> > > > > > > > address
> > > > > > > > any lingering edge cases and put NiFi in a better position to
> > > > > > > > support
> > > > > > > > current Java versions.
> > > > > > > >
> > > > > > > > *JSR 310 for Date and Time Handling*
> > > > > > > >
> > > > > > > > Without making the scope too broad, transitioning internal
> date
> > > > > > > > and time
> > > > > > > > handling to use DateTimeFormatter instead of SimpleDateFormat
> > > > > > > > would provide
> > > > > > > > a number of advantages. The Java Time components provide much
> > > > > > > > better
> > > > > > > > clarity when it comes to handling localized date and time
> > > > > > > > representations,
> > > > > > > > and also avoid the inherent confusion of java.sql.Date
> > extending
> > > > > > > > java.util.Date. Many internal components, specifically
> > > > > > > > Record-oriented
> > > > > > > > processors and services, rely on date parsing, leading to
> > > > > > > > confusion and
> > > > > > > > various workarounds. The pattern formats of SimpleDateFormat
> > and
> > > > > > > > DateTimeFormatter are very similar, but there are a few
> subtle
> > > > > > > > differences.
> > > > > > > > Making this transition would provide a much better foundation
> > > > > > > > going forward.
> > > > > > > > *Conclusion*
> > > > > > > >
> > > > > > > > Thanks for giving this proposal some consideration. Many of
> you
> > > > > > > > have been
> > > > > > > > developing NiFi for years and I look forward to your
> feedback.
> > I
> > > > > > > > would be
> > > > > > > > glad to put together a more formalized recommendation on
> > > > > > > > Confluence and
> > > > > > > > write up Jira epics if this general approach sounds agreeable
> > to
> > > > > > > > the
> > > > > > > > community.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > David Handermann
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by David Handermann <ex...@apache.org>.
Mark,
Thanks for the feedback. It may be better to start a separate thread on
PostHTTP, but can you provide an example flow demonstrating the performance
differences between PostHTTP and InvokeHTTP?
PostHTTP uses the Apache HttpComponents library, whereas InvokeHTTP uses
OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for OkHttp,
so it would be important to test with the most recent version. It is also
worth noting that test classes for PostHTTP are now integration tests as
opposed to unit tests, which means they are not executed as part of the
automated builds.
The NiFi documentation includes the contents of the DeprecationNotice for
PostHTTP:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html
Regards,
David Handermann
On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <ma...@gmail.com> wrote:
> I'm strongly in favor of reducing tech debt, and moving deliberately to a
> 2.0 release. I have a concern with just one processor that is currently
> marked as deprecated: PostHTTP. (I have not evaluated specifically any
> other deprecated components; I cannot say if there are or are not similar
> issues elsewhere.) I understand the rationale for deprecating this
> processor in that it eliminates a processor whose functionality is
> available elsewhere, specifically in the more flexible InvokeHTTP. However,
> in my experience and testing, PostHTTP performs transfers with far greater
> throughput than InvokeHTTP. I would not be in favor of removing PostHTTP
> unless/until InvokeHTTP is refactored to increase its throughput
> capability.
>
> Has anyone else continued to use PostHTTP over InvokeHTTP for similar
> reasons? Or, is there a performance-related configuration for InvokeHTTP I
> may have missed?
>
> Also, in order to help facilitate a smooth transition to 2.0 from a user
> perspective, would it be advisable to add some sort of visual indicator in
> the UI for components that are currently annotated as @Deprecated? This
> might help users proactively modify their flow prior to a release that
> would otherwise break it.
>
>
>
>
> On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <bb...@gmail.com> wrote:
>
> > With the merging of MiNiFi and registry into the NiFi repo, we've
> > actually gone more towards a mono-repo where everything is released
> > together. Eventually it would still be nice to have a smaller base
> > distribution containing just the framework and standard NARs, but it
> > is hard to tackle that until we provide a way for users to easily
> > obtain the NARs in some other way.
> >
> > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <ed...@gmail.com>
> > wrote:
> > >
> > > Given the major version number shift and the spliting up of processors
> > into
> > > multiple NAR's. I'd like to suggest that we start individually
> versioning
> > > individual NARs/ bundles.
> > >
> > > I can see this bringing a large number of benifits including making
> Nifi
> > > more deployable with things RPM, but also potentially allowing those
> that
> > > have to deploy managed Nifi instances easier to upgrade.
> > >
> > > Edward
> > >
> > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, <ot...@gmail.com>
> wrote:
> > >
> > > > The issue with updating the aws sdk is if it breaks any one of the
> > > > processors.
> > > > the Web Gateway API invoke processor for example is not using a high
> > level
> > > > purpose build client and may break.
> > > >
> > > > If we change the aws version, we need to coordinate in such a way
> that
> > they
> > > > all
> > > > can come along reasonably.
> > > > IE: what happens if 1 or 2 break but the rest or OK?
> > > >
> > > >
> > > >
> > > > From: David Handermann <ex...@apache.org>
> > > > <ex...@apache.org>
> > > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> dev@nifi.apache.org>
> > > > Date: July 26, 2021 at 09:33:42
> > > > To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > >
> > > > Chris,
> > > >
> > > > Thanks for the reply and recommendations. It seems like some of the
> > work to
> > > > reorganize the module structure could be done outside of a major
> > release,
> > > > but it would be great to target any breaking changes for 2.0.
> Perhaps a
> > > > separate feature proposal on module restructuring, with the goal of
> > > > supporting optimized builds, would be a helpful way to move that part
> > of
> > > > the discussion forward.
> > > >
> > > > Regarding updating AWS SDK to version 2, it seems like that might be
> > > > possible now. I haven't taken a close look at the referencing
> > components,
> > > > so I'm not sure about the level of effort involved. Minor NiFi
> version
> > > > updates have incorporated major new versions of dependencies. For
> > example,
> > > > NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the
> one
> > > > hand, including the AWS SDK update as part of a major release seems
> > > > helpful, but unless there are changes that break existing component
> > > > properties, upgrading the AWS SDK could be worked independently.
> > Others may
> > > > have more insight into particular usage of that library.
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > > > <ch...@naimuri.com.invalid> wrote:
> > > >
> > > > > Might be worth considering refactoring the build as part of this
> work
> > > > too,
> > > > > e.g. only building the bits of the repo affected by a commit, etc.
> -
> > > > > discussed briefly in previous threads but don't think any changes
> > made
> > > > yet.
> > > > > If NARs/components are likely to be split up and refactored then
> such
> > > > work
> > > > > around the build probably makes sense to consider.
> > > > >
> > > > > I've a couple of PRs open that include updates to Elasticsearch
> > versions
> > > > > already, although I stopped at 7.10.2 (after which Elastic changed
> > > > licence)
> > > > > in case there were licence concerns. But more work can be done to
> > tidy up
> > > > > the processors, absolutely.
> > > > >
> > > > > AWS libraries to v2 would seem a sensible move and a refactor of
> > those
> > > > > processors as well.
> > > > >
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris Sampson
> > > > >
> > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > > > exceptionfactory@apache.org>
> > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks for pointing out the standard NAR bundles, Mark. There
> are a
> > > > > number
> > > > > > of components in the standard NAR bundles with particular
> > dependencies
> > > > > that
> > > > > > would make more sense in separate NARs. Reorganizing the standard
> > NAR
> > > > to
> > > > > > components with limited dependencies and wide applicability would
> > > > > > definitely help with future maintenance.
> > > > > >
> > > > > > Regards,
> > > > > > David Handermann
> > > > > >
> > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <
> markap14@hotmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > There’s also some code that exists in order to maintain
> backward
> > > > > > > compatibility in the repositories. I would very much like the
> > > > > > repositories
> > > > > > > to contain no unnecessary code. And swap file format supports
> > really
> > > > > old
> > > > > > > formats. And the old impls of the repositories themselves, like
> > > > > > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of
> > stuff
> > > > > > there
> > > > > > > that can be removed. And some methods in ProcessSession that
> are
> > > > never
> > > > > > used
> > > > > > > by any processor in the codebase but exists in the public API
> so
> > > > can’t
> > > > > be
> > > > > > > removed till 2.0.
> > > > > > >
> > > > > > > I think his is also a great time to clean up the “standard
> nar.”
> > At
> > > > > this
> > > > > > > point, it’s something like 70 MB. And many of the components
> > there
> > > > are
> > > > > > not
> > > > > > > really “standard” - things like connecting to FTP & SFTP
> > servers, XML
> > > > > > > processing, Jolt transform, etc. could potentially be moved
> into
> > > > other
> > > > > > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is
> > 6.9 MB
> > > > is
> > > > > > not
> > > > > > > necessary for stateless or minifi java. Lots of things probably
> > to
> > > > > > > reconsider within the standard nar.
> > > > > > >
> > > > > > > I definitely think this is a reasonable approach, to allow for
> a
> > 2.0
> > > > > that
> > > > > > > is not a huge feature release but allows the project to be
> > simpler
> > > > and
> > > > > > more
> > > > > > > nimble.
> > > > > > >
> > > > > > > Thanks
> > > > > > > -Mark
> > > > > > >
> > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> > mikerthomsen@gmail.com
> > > > > > <mailto:
> > > > > > > mikerthomsen@gmail.com>> wrote:
> > > > > > >
> > > > > > > Russell,
> > > > > > >
> > > > > > > AFAICT from looking at Elastic's repos, the low level REST
> > client is
> > > > > > > still fine.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> >
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > > > > >
> > > > > > > Our Elasticsearch support is spread over two NARs at present.
> One
> > > > uses
> > > > > > > OkHttp and the other uses that low level Elastic REST client.
> > > > > > > Therefore, I think we're fine on licensing for the moment.
> > > > > > >
> > > > > > > Mike
> > > > > > >
> > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > > > russ@windofkeltia.com
> > > > > > > <ma...@windofkeltia.com>> wrote:
> > > > > > >
> > > > > > > Bringing up Elastic also reminds me that the Elastic framework
> > has
> > > > just
> > > > > > > recently transitioned out of Open Source, so to acknowledge
> that,
> > > > maybe
> > > > > > > some effort toward OpenSearch--I say this not understanding
> > exactly
> > > > how
> > > > > > > this sort of thing is considered in a large-scale, world-class
> > > > software
> > > > > > > project like Apache NiFi. (I'm not a contributor, just a
> grateful
> > > > > > > consumer.)
> > > > > > >
> > > > > > > Russ
> > > > > > >
> > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > > > > > Along with the itemized list for ancient components we should
> > look at
> > > > > > > updating versions of drivers, SDKs, etc. for external systems
> > such as
> > > > > > > Elasticsearch, Cassandra, etc. There may be breaking changes
> but
> > 2.0
> > > > > > > is probably the right time to get things up to date to make
> them
> > more
> > > > > > > useful to more people.
> > > > > > >
> > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> > thenatog@gmail.com
> > > > > > <mailto:
> > > > > > > thenatog@gmail.com>> wrote:
> > > > > > > I'm a +1 for removing pretty much all of this stuff. There are
> > > > security
> > > > > > > implications to keeping old dependencies around, so the more
> old
> > code
> > > > > we
> > > > > > > can remove the better. I agree that eventually we need to move
> to
> > > > > > > supporting only Java 11+, and as our next release will probably
> > be
> > > > > about
> > > > > > 4
> > > > > > > - 6 months from now that doesn't seem too soon. We could
> > potentially
> > > > > > break
> > > > > > > this in two and remove the deprecated processors and leave 1.x
> on
> > > > Java
> > > > > 8,
> > > > > > > and finally start on 2.x which would support only Java 11. I'm
> > unsure
> > > > > of
> > > > > > > what implications changing the date and time handling would
> have
> > -
> > > > for
> > > > > > > running systems that use long term historical logs, unexpected
> > > > impacts
> > > > > to
> > > > > > > time logging could be a problem.
> > > > > > >
> > > > > > > As Joe says I think feature work will have to be dedicated to
> > 2.x and
> > > > > we
> > > > > > > could support 1.x for security fixes for some period of time.
> 2.x
> > > > seems
> > > > > > > like a gargantuan task but it's probably time to get started.
> Not
> > > > sure
> > > > > > how
> > > > > > > we handle all open PRs and the transition between 1.x and 2.x.
> > > > > > >
> > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <joe.witt@gmail.com
> > > > <mailto:
> > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > >
> > > > > > > Jon
> > > > > > >
> > > > > > > You're right we have to be careful and you're right there are
> > still
> > > > > > > significant Java 8 users out there. But we also have to be
> > careful
> > > > > > > about security and sustainability of the codebase. If we had
> > talked
> > > > > > > about this last year when that article came out I'd have agreed
> > it is
> > > > > > > too early. Interestingly that link seems to get updated and I
> > tried
> > > > > > > [1] and found more recent data (not sure how recent). Anyway it
> > > > > > > suggests Java 8 is still the top dog but we see good growth on
> > 11. In
> > > > > > > my $dayjob this aligns to what I'm seeing too. Customers didn't
> > seem
> > > > > > > to care about Java 11 until later half last year and now
> > suddenly it
> > > > > > > is all over the place.
> > > > > > >
> > > > > > > I think once we put out a NiFi 2.0 release we'd see rapid
> > decrease in
> > > > > > > work on the 1.x line just being blunt. We did this many years
> ago
> > > > > > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a
> > year or
> > > > > > > so) but it was purely bug fix/security related bits. We would
> > need to
> > > > > > > do something similar. But feature work would almost certainly
> go
> > to
> > > > > > > the 2.x line. Maybe there are other workable models but my
> > instinct
> > > > > > > suggests this is likely to follow a similar path.
> > > > > > >
> > > > > > > ...anyway I agree it isn't that easy of a call to dump Java 8.
> We
> > > > > > > need to make the call in both the interests of the user base
> and
> > the
> > > > > > > contributor base of the community.
> > > > > > >
> > > > > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > > > > >
> > > > > > >
> > > > > > > Thanks
> > > > > > > Joe
> > > > > > >
> > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <joe.witt@gmail.com
> > <mailto:
> > > > > > > joe.witt@gmail.com>> wrote:
> > > > > > > Russ
> > > > > > >
> > > > > > > Yeah the flow registry is a key part of it. But also now you
> can
> > > > > > > download the flow definition in JSON (upload i think is there
> now
> > > > > > > too). Templates offered a series of challenges such as we store
> > them
> > > > > > > in the flow definition which has made flows massive in an
> > unintended
> > > > > > > way which isn't fun for cluster behavior.
> > > > > > >
> > > > > > > We have a couple cases where we headed down a particular
> concept
> > and
> > > > > > > came up with better approaches later. We need to reconcile
> these
> > with
> > > > > > > the benefit of hindsight, and while being careful to be not
> > overly
> > > > > > > disruptive to existing users, to reduce the
> codebase/maintenance
> > > > > > > burden and allow continued evolution of the project.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > > > russ@windofkeltia.com
> > > > > > > <ma...@windofkeltia.com>>
> > > > > > > wrote:
> > > > > > > Joe,
> > > > > > >
> > > > > > > I apologize for the off-topic intrusion, but what replaces
> > templates?
> > > > > > > The Registry? Templates rocked and we have used them since
> 0.5.x.
> > > > > > >
> > > > > > > Russ
> > > > > > >
> > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > > > David,
> > > > > > >
> > > > > > > I think this is a highly reasonable approach and such a focus
> > will
> > > > > > > greatly help make a 2.0 release far more approachable to knock
> > out.
> > > > > > > Not only that but tech debt reduction would help make work
> > towards
> > > > > > > major features we'd think about in a 'major release' sense more
> > > > > > > approachable.
> > > > > > >
> > > > > > > We should remove all deprecated things (as well as verify we
> > have the
> > > > > > > right list). We should remove/consider removal of deprecated
> > > > > > > concepts
> > > > > > > like templates. We should consider whether we can resolve the
> > > > > > > various
> > > > > > > ways we've handled what are now parameters down to one clean
> > > > > > > approach.
> > > > > > > We should remove options in the nifi.properties which turn out
> to
> > > > > > > never be used quite right (if there are). There is quite a bit
> we
> > > > > > > can
> > > > > > > do purely in the name of tech debt reduction.
> > > > > > >
> > > > > > > Lots to consider here but I think this is the right discussion.
> > > > > > >
> > > > > > > Than ks
> > > > > > >
> > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bbende@gmail.com
> > > > <mailto:
> > > > > > > bbende@gmail.com>>
> > > > > > > wrote:
> > > > > > > I'm a +1 for this... Not sure if this falls under "Removing
> > > > > > > Deprecated
> > > > > > > Components", but I think we should also look at anything that
> has
> > > > > > > been
> > > > > > > marked as deprecated throughout the code base as a candidate
> for
> > > > > > > removal. There are quite a few classes, methods, properties,
> etc
> > > > > > > that
> > > > > > > have been waiting for a chance to be removed.
> > > > > > >
> > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > > > > <exceptionfactory@apache.org<mailto:
> exceptionfactory@apache.org
> > >>
> > > > > wrote:
> > > > > > > Team,
> > > > > > >
> > > > > > > With all of the excellent work that many have contributed to
> NiFi
> > > > > > > over the
> > > > > > > years, the code base has also accumulated some amount of
> > technical
> > > > > > > debt. A
> > > > > > > handful of components have been marked as deprecated, and some
> > > > > > > components
> > > > > > > remain in the code base to support integration with old
> versions
> > > > > > > of various
> > > > > > > products. Following the principles of semantic versioning,
> > > > > > > introducing a
> > > > > > > major release would provide the opportunity to remove these
> > > > > > > deprecated and
> > > > > > > unsupported components.
> > > > > > >
> > > > > > > Rather than focusing the next major release on new features,
> what
> > > > > > > do you
> > > > > > > think about focusing on technical debt removal? This approach
> > > > > > > would not
> > > > > > > make for the most interesting release, but it provides the
> > > > > > > opportunity to
> > > > > > > clean up elements that involve breaking changes.
> > > > > > >
> > > > > > > Focusing on technical debt, at least three primary goals come
> to
> > > > > > > mind for
> > > > > > > the next major release:
> > > > > > >
> > > > > > > 1. Removal of deprecated and unmaintained components
> > > > > > > 2. Require Java 11 as the minimum supported version
> > > > > > > 3. Transition internal date and time handling to JSR 310
> > java.time
> > > > > > > components
> > > > > > >
> > > > > > > *Removing Deprecated Components*
> > > > > > >
> > > > > > > Removing support for older and deprecated components provides a
> > > > > > > great
> > > > > > > opportunity to improve the overall security posture when it
> comes
> > > > > > > to
> > > > > > > maintaining dependencies. The OWASP dependency plugin report
> > > > > > > currently
> > > > > > > generates 50 MB of HTML for questionable dependencies, many of
> > > > > > > which are
> > > > > > > related to old versions of various libraries.
> > > > > > >
> > > > > > > As a starting point, here are a handful of components and
> > > > > > > extension modules
> > > > > > > that could be targeted for removal in a major version:
> > > > > > >
> > > > > > > - PostHTTP and GetHTTP
> > > > > > > - ListenLumberjack and the entire nifi-lumberjack-bundle
> > > > > > > - ListenBeats and the entire nifi-beats-bundle
> > > > > > > - Elasticsearch 5 components
> > > > > > > - Hive 1 and 2 components
> > > > > > >
> > > > > > > *Requiring Java 11*
> > > > > > >
> > > > > > > Java 8 is now over seven years old, and NiFi has supported
> > general
> > > > > > > compatibility with Java 11 for several years. NiFi 1.14.0
> > > > > > > incorporated
> > > > > > > internal improvements specifically related to TLS 1.3, which
> > > > > > > allowed
> > > > > > > closing out the long-running Java 11 compatibility epic
> > NIFI-5174.
> > > > > > > Making
> > > > > > > Java 11 the minimum required version provides the opportunity
> to
> > > > > > > address
> > > > > > > any lingering edge cases and put NiFi in a better position to
> > > > > > > support
> > > > > > > current Java versions.
> > > > > > >
> > > > > > > *JSR 310 for Date and Time Handling*
> > > > > > >
> > > > > > > Without making the scope too broad, transitioning internal date
> > > > > > > and time
> > > > > > > handling to use DateTimeFormatter instead of SimpleDateFormat
> > > > > > > would provide
> > > > > > > a number of advantages. The Java Time components provide much
> > > > > > > better
> > > > > > > clarity when it comes to handling localized date and time
> > > > > > > representations,
> > > > > > > and also avoid the inherent confusion of java.sql.Date
> extending
> > > > > > > java.util.Date. Many internal components, specifically
> > > > > > > Record-oriented
> > > > > > > processors and services, rely on date parsing, leading to
> > > > > > > confusion and
> > > > > > > various workarounds. The pattern formats of SimpleDateFormat
> and
> > > > > > > DateTimeFormatter are very similar, but there are a few subtle
> > > > > > > differences.
> > > > > > > Making this transition would provide a much better foundation
> > > > > > > going forward.
> > > > > > > *Conclusion*
> > > > > > >
> > > > > > > Thanks for giving this proposal some consideration. Many of you
> > > > > > > have been
> > > > > > > developing NiFi for years and I look forward to your feedback.
> I
> > > > > > > would be
> > > > > > > glad to put together a more formalized recommendation on
> > > > > > > Confluence and
> > > > > > > write up Jira epics if this general approach sounds agreeable
> to
> > > > > > > the
> > > > > > > community.
> > > > > > >
> > > > > > > Regards,
> > > > > > > David Handermann
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Mark Bean <ma...@gmail.com>.
I'm strongly in favor of reducing tech debt, and moving deliberately to a
2.0 release. I have a concern with just one processor that is currently
marked as deprecated: PostHTTP. (I have not evaluated specifically any
other deprecated components; I cannot say if there are or are not similar
issues elsewhere.) I understand the rationale for deprecating this
processor in that it eliminates a processor whose functionality is
available elsewhere, specifically in the more flexible InvokeHTTP. However,
in my experience and testing, PostHTTP performs transfers with far greater
throughput than InvokeHTTP. I would not be in favor of removing PostHTTP
unless/until InvokeHTTP is refactored to increase its throughput capability.
Has anyone else continued to use PostHTTP over InvokeHTTP for similar
reasons? Or, is there a performance-related configuration for InvokeHTTP I
may have missed?
Also, in order to help facilitate a smooth transition to 2.0 from a user
perspective, would it be advisable to add some sort of visual indicator in
the UI for components that are currently annotated as @Deprecated? This
might help users proactively modify their flow prior to a release that
would otherwise break it.
On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <bb...@gmail.com> wrote:
> With the merging of MiNiFi and registry into the NiFi repo, we've
> actually gone more towards a mono-repo where everything is released
> together. Eventually it would still be nice to have a smaller base
> distribution containing just the framework and standard NARs, but it
> is hard to tackle that until we provide a way for users to easily
> obtain the NARs in some other way.
>
> On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <ed...@gmail.com>
> wrote:
> >
> > Given the major version number shift and the spliting up of processors
> into
> > multiple NAR's. I'd like to suggest that we start individually versioning
> > individual NARs/ bundles.
> >
> > I can see this bringing a large number of benifits including making Nifi
> > more deployable with things RPM, but also potentially allowing those that
> > have to deploy managed Nifi instances easier to upgrade.
> >
> > Edward
> >
> > On Mon, 26 Jul 2021, 20:42 Otto Fowler, <ot...@gmail.com> wrote:
> >
> > > The issue with updating the aws sdk is if it breaks any one of the
> > > processors.
> > > the Web Gateway API invoke processor for example is not using a high
> level
> > > purpose build client and may break.
> > >
> > > If we change the aws version, we need to coordinate in such a way that
> they
> > > all
> > > can come along reasonably.
> > > IE: what happens if 1 or 2 break but the rest or OK?
> > >
> > >
> > >
> > > From: David Handermann <ex...@apache.org>
> > > <ex...@apache.org>
> > > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > > Date: July 26, 2021 at 09:33:42
> > > To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > >
> > > Chris,
> > >
> > > Thanks for the reply and recommendations. It seems like some of the
> work to
> > > reorganize the module structure could be done outside of a major
> release,
> > > but it would be great to target any breaking changes for 2.0. Perhaps a
> > > separate feature proposal on module restructuring, with the goal of
> > > supporting optimized builds, would be a helpful way to move that part
> of
> > > the discussion forward.
> > >
> > > Regarding updating AWS SDK to version 2, it seems like that might be
> > > possible now. I haven't taken a close look at the referencing
> components,
> > > so I'm not sure about the level of effort involved. Minor NiFi version
> > > updates have incorporated major new versions of dependencies. For
> example,
> > > NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the one
> > > hand, including the AWS SDK update as part of a major release seems
> > > helpful, but unless there are changes that break existing component
> > > properties, upgrading the AWS SDK could be worked independently.
> Others may
> > > have more insight into particular usage of that library.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > > <ch...@naimuri.com.invalid> wrote:
> > >
> > > > Might be worth considering refactoring the build as part of this work
> > > too,
> > > > e.g. only building the bits of the repo affected by a commit, etc. -
> > > > discussed briefly in previous threads but don't think any changes
> made
> > > yet.
> > > > If NARs/components are likely to be split up and refactored then such
> > > work
> > > > around the build probably makes sense to consider.
> > > >
> > > > I've a couple of PRs open that include updates to Elasticsearch
> versions
> > > > already, although I stopped at 7.10.2 (after which Elastic changed
> > > licence)
> > > > in case there were licence concerns. But more work can be done to
> tidy up
> > > > the processors, absolutely.
> > > >
> > > > AWS libraries to v2 would seem a sensible move and a refactor of
> those
> > > > processors as well.
> > > >
> > > >
> > > > Cheers,
> > > >
> > > > Chris Sampson
> > > >
> > > > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > > exceptionfactory@apache.org>
> > >
> > > > wrote:
> > > >
> > > > > Thanks for pointing out the standard NAR bundles, Mark. There are a
> > > > number
> > > > > of components in the standard NAR bundles with particular
> dependencies
> > > > that
> > > > > would make more sense in separate NARs. Reorganizing the standard
> NAR
> > > to
> > > > > components with limited dependencies and wide applicability would
> > > > > definitely help with future maintenance.
> > > > >
> > > > > Regards,
> > > > > David Handermann
> > > > >
> > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <ma...@hotmail.com>
> > > > wrote:
> > > > >
> > > > > > There’s also some code that exists in order to maintain backward
> > > > > > compatibility in the repositories. I would very much like the
> > > > > repositories
> > > > > > to contain no unnecessary code. And swap file format supports
> really
> > > > old
> > > > > > formats. And the old impls of the repositories themselves, like
> > > > > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of
> stuff
> > > > > there
> > > > > > that can be removed. And some methods in ProcessSession that are
> > > never
> > > > > used
> > > > > > by any processor in the codebase but exists in the public API so
> > > can’t
> > > > be
> > > > > > removed till 2.0.
> > > > > >
> > > > > > I think his is also a great time to clean up the “standard nar.”
> At
> > > > this
> > > > > > point, it’s something like 70 MB. And many of the components
> there
> > > are
> > > > > not
> > > > > > really “standard” - things like connecting to FTP & SFTP
> servers, XML
> > > > > > processing, Jolt transform, etc. could potentially be moved into
> > > other
> > > > > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is
> 6.9 MB
> > > is
> > > > > not
> > > > > > necessary for stateless or minifi java. Lots of things probably
> to
> > > > > > reconsider within the standard nar.
> > > > > >
> > > > > > I definitely think this is a reasonable approach, to allow for a
> 2.0
> > > > that
> > > > > > is not a huge feature release but allows the project to be
> simpler
> > > and
> > > > > more
> > > > > > nimble.
> > > > > >
> > > > > > Thanks
> > > > > > -Mark
> > > > > >
> > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <
> mikerthomsen@gmail.com
> > > > > <mailto:
> > > > > > mikerthomsen@gmail.com>> wrote:
> > > > > >
> > > > > > Russell,
> > > > > >
> > > > > > AFAICT from looking at Elastic's repos, the low level REST
> client is
> > > > > > still fine.
> > > > > >
> > > > >
> > > >
> > >
> > >
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > > > >
> > > > > > Our Elasticsearch support is spread over two NARs at present. One
> > > uses
> > > > > > OkHttp and the other uses that low level Elastic REST client.
> > > > > > Therefore, I think we're fine on licensing for the moment.
> > > > > >
> > > > > > Mike
> > > > > >
> > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > > russ@windofkeltia.com
> > > > > > <ma...@windofkeltia.com>> wrote:
> > > > > >
> > > > > > Bringing up Elastic also reminds me that the Elastic framework
> has
> > > just
> > > > > > recently transitioned out of Open Source, so to acknowledge that,
> > > maybe
> > > > > > some effort toward OpenSearch--I say this not understanding
> exactly
> > > how
> > > > > > this sort of thing is considered in a large-scale, world-class
> > > software
> > > > > > project like Apache NiFi. (I'm not a contributor, just a grateful
> > > > > > consumer.)
> > > > > >
> > > > > > Russ
> > > > > >
> > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > > > > Along with the itemized list for ancient components we should
> look at
> > > > > > updating versions of drivers, SDKs, etc. for external systems
> such as
> > > > > > Elasticsearch, Cassandra, etc. There may be breaking changes but
> 2.0
> > > > > > is probably the right time to get things up to date to make them
> more
> > > > > > useful to more people.
> > > > > >
> > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <
> thenatog@gmail.com
> > > > > <mailto:
> > > > > > thenatog@gmail.com>> wrote:
> > > > > > I'm a +1 for removing pretty much all of this stuff. There are
> > > security
> > > > > > implications to keeping old dependencies around, so the more old
> code
> > > > we
> > > > > > can remove the better. I agree that eventually we need to move to
> > > > > > supporting only Java 11+, and as our next release will probably
> be
> > > > about
> > > > > 4
> > > > > > - 6 months from now that doesn't seem too soon. We could
> potentially
> > > > > break
> > > > > > this in two and remove the deprecated processors and leave 1.x on
> > > Java
> > > > 8,
> > > > > > and finally start on 2.x which would support only Java 11. I'm
> unsure
> > > > of
> > > > > > what implications changing the date and time handling would have
> -
> > > for
> > > > > > running systems that use long term historical logs, unexpected
> > > impacts
> > > > to
> > > > > > time logging could be a problem.
> > > > > >
> > > > > > As Joe says I think feature work will have to be dedicated to
> 2.x and
> > > > we
> > > > > > could support 1.x for security fixes for some period of time. 2.x
> > > seems
> > > > > > like a gargantuan task but it's probably time to get started. Not
> > > sure
> > > > > how
> > > > > > we handle all open PRs and the transition between 1.x and 2.x.
> > > > > >
> > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <joe.witt@gmail.com
> > > <mailto:
> > > > > > joe.witt@gmail.com>> wrote:
> > > > > >
> > > > > > Jon
> > > > > >
> > > > > > You're right we have to be careful and you're right there are
> still
> > > > > > significant Java 8 users out there. But we also have to be
> careful
> > > > > > about security and sustainability of the codebase. If we had
> talked
> > > > > > about this last year when that article came out I'd have agreed
> it is
> > > > > > too early. Interestingly that link seems to get updated and I
> tried
> > > > > > [1] and found more recent data (not sure how recent). Anyway it
> > > > > > suggests Java 8 is still the top dog but we see good growth on
> 11. In
> > > > > > my $dayjob this aligns to what I'm seeing too. Customers didn't
> seem
> > > > > > to care about Java 11 until later half last year and now
> suddenly it
> > > > > > is all over the place.
> > > > > >
> > > > > > I think once we put out a NiFi 2.0 release we'd see rapid
> decrease in
> > > > > > work on the 1.x line just being blunt. We did this many years ago
> > > > > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a
> year or
> > > > > > so) but it was purely bug fix/security related bits. We would
> need to
> > > > > > do something similar. But feature work would almost certainly go
> to
> > > > > > the 2.x line. Maybe there are other workable models but my
> instinct
> > > > > > suggests this is likely to follow a similar path.
> > > > > >
> > > > > > ...anyway I agree it isn't that easy of a call to dump Java 8. We
> > > > > > need to make the call in both the interests of the user base and
> the
> > > > > > contributor base of the community.
> > > > > >
> > > > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > > > >
> > > > > >
> > > > > > Thanks
> > > > > > Joe
> > > > > >
> > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <joe.witt@gmail.com
> <mailto:
> > > > > > joe.witt@gmail.com>> wrote:
> > > > > > Russ
> > > > > >
> > > > > > Yeah the flow registry is a key part of it. But also now you can
> > > > > > download the flow definition in JSON (upload i think is there now
> > > > > > too). Templates offered a series of challenges such as we store
> them
> > > > > > in the flow definition which has made flows massive in an
> unintended
> > > > > > way which isn't fun for cluster behavior.
> > > > > >
> > > > > > We have a couple cases where we headed down a particular concept
> and
> > > > > > came up with better approaches later. We need to reconcile these
> with
> > > > > > the benefit of hindsight, and while being careful to be not
> overly
> > > > > > disruptive to existing users, to reduce the codebase/maintenance
> > > > > > burden and allow continued evolution of the project.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > > russ@windofkeltia.com
> > > > > > <ma...@windofkeltia.com>>
> > > > > > wrote:
> > > > > > Joe,
> > > > > >
> > > > > > I apologize for the off-topic intrusion, but what replaces
> templates?
> > > > > > The Registry? Templates rocked and we have used them since 0.5.x.
> > > > > >
> > > > > > Russ
> > > > > >
> > > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > > David,
> > > > > >
> > > > > > I think this is a highly reasonable approach and such a focus
> will
> > > > > > greatly help make a 2.0 release far more approachable to knock
> out.
> > > > > > Not only that but tech debt reduction would help make work
> towards
> > > > > > major features we'd think about in a 'major release' sense more
> > > > > > approachable.
> > > > > >
> > > > > > We should remove all deprecated things (as well as verify we
> have the
> > > > > > right list). We should remove/consider removal of deprecated
> > > > > > concepts
> > > > > > like templates. We should consider whether we can resolve the
> > > > > > various
> > > > > > ways we've handled what are now parameters down to one clean
> > > > > > approach.
> > > > > > We should remove options in the nifi.properties which turn out to
> > > > > > never be used quite right (if there are). There is quite a bit we
> > > > > > can
> > > > > > do purely in the name of tech debt reduction.
> > > > > >
> > > > > > Lots to consider here but I think this is the right discussion.
> > > > > >
> > > > > > Than ks
> > > > > >
> > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bbende@gmail.com
> > > <mailto:
> > > > > > bbende@gmail.com>>
> > > > > > wrote:
> > > > > > I'm a +1 for this... Not sure if this falls under "Removing
> > > > > > Deprecated
> > > > > > Components", but I think we should also look at anything that has
> > > > > > been
> > > > > > marked as deprecated throughout the code base as a candidate for
> > > > > > removal. There are quite a few classes, methods, properties, etc
> > > > > > that
> > > > > > have been waiting for a chance to be removed.
> > > > > >
> > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > > > <exceptionfactory@apache.org<mailto:exceptionfactory@apache.org
> >>
> > > > wrote:
> > > > > > Team,
> > > > > >
> > > > > > With all of the excellent work that many have contributed to NiFi
> > > > > > over the
> > > > > > years, the code base has also accumulated some amount of
> technical
> > > > > > debt. A
> > > > > > handful of components have been marked as deprecated, and some
> > > > > > components
> > > > > > remain in the code base to support integration with old versions
> > > > > > of various
> > > > > > products. Following the principles of semantic versioning,
> > > > > > introducing a
> > > > > > major release would provide the opportunity to remove these
> > > > > > deprecated and
> > > > > > unsupported components.
> > > > > >
> > > > > > Rather than focusing the next major release on new features, what
> > > > > > do you
> > > > > > think about focusing on technical debt removal? This approach
> > > > > > would not
> > > > > > make for the most interesting release, but it provides the
> > > > > > opportunity to
> > > > > > clean up elements that involve breaking changes.
> > > > > >
> > > > > > Focusing on technical debt, at least three primary goals come to
> > > > > > mind for
> > > > > > the next major release:
> > > > > >
> > > > > > 1. Removal of deprecated and unmaintained components
> > > > > > 2. Require Java 11 as the minimum supported version
> > > > > > 3. Transition internal date and time handling to JSR 310
> java.time
> > > > > > components
> > > > > >
> > > > > > *Removing Deprecated Components*
> > > > > >
> > > > > > Removing support for older and deprecated components provides a
> > > > > > great
> > > > > > opportunity to improve the overall security posture when it comes
> > > > > > to
> > > > > > maintaining dependencies. The OWASP dependency plugin report
> > > > > > currently
> > > > > > generates 50 MB of HTML for questionable dependencies, many of
> > > > > > which are
> > > > > > related to old versions of various libraries.
> > > > > >
> > > > > > As a starting point, here are a handful of components and
> > > > > > extension modules
> > > > > > that could be targeted for removal in a major version:
> > > > > >
> > > > > > - PostHTTP and GetHTTP
> > > > > > - ListenLumberjack and the entire nifi-lumberjack-bundle
> > > > > > - ListenBeats and the entire nifi-beats-bundle
> > > > > > - Elasticsearch 5 components
> > > > > > - Hive 1 and 2 components
> > > > > >
> > > > > > *Requiring Java 11*
> > > > > >
> > > > > > Java 8 is now over seven years old, and NiFi has supported
> general
> > > > > > compatibility with Java 11 for several years. NiFi 1.14.0
> > > > > > incorporated
> > > > > > internal improvements specifically related to TLS 1.3, which
> > > > > > allowed
> > > > > > closing out the long-running Java 11 compatibility epic
> NIFI-5174.
> > > > > > Making
> > > > > > Java 11 the minimum required version provides the opportunity to
> > > > > > address
> > > > > > any lingering edge cases and put NiFi in a better position to
> > > > > > support
> > > > > > current Java versions.
> > > > > >
> > > > > > *JSR 310 for Date and Time Handling*
> > > > > >
> > > > > > Without making the scope too broad, transitioning internal date
> > > > > > and time
> > > > > > handling to use DateTimeFormatter instead of SimpleDateFormat
> > > > > > would provide
> > > > > > a number of advantages. The Java Time components provide much
> > > > > > better
> > > > > > clarity when it comes to handling localized date and time
> > > > > > representations,
> > > > > > and also avoid the inherent confusion of java.sql.Date extending
> > > > > > java.util.Date. Many internal components, specifically
> > > > > > Record-oriented
> > > > > > processors and services, rely on date parsing, leading to
> > > > > > confusion and
> > > > > > various workarounds. The pattern formats of SimpleDateFormat and
> > > > > > DateTimeFormatter are very similar, but there are a few subtle
> > > > > > differences.
> > > > > > Making this transition would provide a much better foundation
> > > > > > going forward.
> > > > > > *Conclusion*
> > > > > >
> > > > > > Thanks for giving this proposal some consideration. Many of you
> > > > > > have been
> > > > > > developing NiFi for years and I look forward to your feedback. I
> > > > > > would be
> > > > > > glad to put together a more formalized recommendation on
> > > > > > Confluence and
> > > > > > write up Jira epics if this general approach sounds agreeable to
> > > > > > the
> > > > > > community.
> > > > > >
> > > > > > Regards,
> > > > > > David Handermann
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Bryan Bende <bb...@gmail.com>.
With the merging of MiNiFi and registry into the NiFi repo, we've
actually gone more towards a mono-repo where everything is released
together. Eventually it would still be nice to have a smaller base
distribution containing just the framework and standard NARs, but it
is hard to tackle that until we provide a way for users to easily
obtain the NARs in some other way.
On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <ed...@gmail.com> wrote:
>
> Given the major version number shift and the spliting up of processors into
> multiple NAR's. I'd like to suggest that we start individually versioning
> individual NARs/ bundles.
>
> I can see this bringing a large number of benifits including making Nifi
> more deployable with things RPM, but also potentially allowing those that
> have to deploy managed Nifi instances easier to upgrade.
>
> Edward
>
> On Mon, 26 Jul 2021, 20:42 Otto Fowler, <ot...@gmail.com> wrote:
>
> > The issue with updating the aws sdk is if it breaks any one of the
> > processors.
> > the Web Gateway API invoke processor for example is not using a high level
> > purpose build client and may break.
> >
> > If we change the aws version, we need to coordinate in such a way that they
> > all
> > can come along reasonably.
> > IE: what happens if 1 or 2 break but the rest or OK?
> >
> >
> >
> > From: David Handermann <ex...@apache.org>
> > <ex...@apache.org>
> > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > Date: July 26, 2021 at 09:33:42
> > To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> >
> > Chris,
> >
> > Thanks for the reply and recommendations. It seems like some of the work to
> > reorganize the module structure could be done outside of a major release,
> > but it would be great to target any breaking changes for 2.0. Perhaps a
> > separate feature proposal on module restructuring, with the goal of
> > supporting optimized builds, would be a helpful way to move that part of
> > the discussion forward.
> >
> > Regarding updating AWS SDK to version 2, it seems like that might be
> > possible now. I haven't taken a close look at the referencing components,
> > so I'm not sure about the level of effort involved. Minor NiFi version
> > updates have incorporated major new versions of dependencies. For example,
> > NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the one
> > hand, including the AWS SDK update as part of a major release seems
> > helpful, but unless there are changes that break existing component
> > properties, upgrading the AWS SDK could be worked independently. Others may
> > have more insight into particular usage of that library.
> >
> > Regards,
> > David Handermann
> >
> > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> > <ch...@naimuri.com.invalid> wrote:
> >
> > > Might be worth considering refactoring the build as part of this work
> > too,
> > > e.g. only building the bits of the repo affected by a commit, etc. -
> > > discussed briefly in previous threads but don't think any changes made
> > yet.
> > > If NARs/components are likely to be split up and refactored then such
> > work
> > > around the build probably makes sense to consider.
> > >
> > > I've a couple of PRs open that include updates to Elasticsearch versions
> > > already, although I stopped at 7.10.2 (after which Elastic changed
> > licence)
> > > in case there were licence concerns. But more work can be done to tidy up
> > > the processors, absolutely.
> > >
> > > AWS libraries to v2 would seem a sensible move and a refactor of those
> > > processors as well.
> > >
> > >
> > > Cheers,
> > >
> > > Chris Sampson
> > >
> > > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > exceptionfactory@apache.org>
> >
> > > wrote:
> > >
> > > > Thanks for pointing out the standard NAR bundles, Mark. There are a
> > > number
> > > > of components in the standard NAR bundles with particular dependencies
> > > that
> > > > would make more sense in separate NARs. Reorganizing the standard NAR
> > to
> > > > components with limited dependencies and wide applicability would
> > > > definitely help with future maintenance.
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <ma...@hotmail.com>
> > > wrote:
> > > >
> > > > > There’s also some code that exists in order to maintain backward
> > > > > compatibility in the repositories. I would very much like the
> > > > repositories
> > > > > to contain no unnecessary code. And swap file format supports really
> > > old
> > > > > formats. And the old impls of the repositories themselves, like
> > > > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff
> > > > there
> > > > > that can be removed. And some methods in ProcessSession that are
> > never
> > > > used
> > > > > by any processor in the codebase but exists in the public API so
> > can’t
> > > be
> > > > > removed till 2.0.
> > > > >
> > > > > I think his is also a great time to clean up the “standard nar.” At
> > > this
> > > > > point, it’s something like 70 MB. And many of the components there
> > are
> > > > not
> > > > > really “standard” - things like connecting to FTP & SFTP servers, XML
> > > > > processing, Jolt transform, etc. could potentially be moved into
> > other
> > > > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is 6.9 MB
> > is
> > > > not
> > > > > necessary for stateless or minifi java. Lots of things probably to
> > > > > reconsider within the standard nar.
> > > > >
> > > > > I definitely think this is a reasonable approach, to allow for a 2.0
> > > that
> > > > > is not a huge feature release but allows the project to be simpler
> > and
> > > > more
> > > > > nimble.
> > > > >
> > > > > Thanks
> > > > > -Mark
> > > > >
> > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <mikerthomsen@gmail.com
> > > > <mailto:
> > > > > mikerthomsen@gmail.com>> wrote:
> > > > >
> > > > > Russell,
> > > > >
> > > > > AFAICT from looking at Elastic's repos, the low level REST client is
> > > > > still fine.
> > > > >
> > > >
> > >
> >
> > https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > > >
> > > > > Our Elasticsearch support is spread over two NARs at present. One
> > uses
> > > > > OkHttp and the other uses that low level Elastic REST client.
> > > > > Therefore, I think we're fine on licensing for the moment.
> > > > >
> > > > > Mike
> > > > >
> > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> > russ@windofkeltia.com
> > > > > <ma...@windofkeltia.com>> wrote:
> > > > >
> > > > > Bringing up Elastic also reminds me that the Elastic framework has
> > just
> > > > > recently transitioned out of Open Source, so to acknowledge that,
> > maybe
> > > > > some effort toward OpenSearch--I say this not understanding exactly
> > how
> > > > > this sort of thing is considered in a large-scale, world-class
> > software
> > > > > project like Apache NiFi. (I'm not a contributor, just a grateful
> > > > > consumer.)
> > > > >
> > > > > Russ
> > > > >
> > > > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > > > Along with the itemized list for ancient components we should look at
> > > > > updating versions of drivers, SDKs, etc. for external systems such as
> > > > > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0
> > > > > is probably the right time to get things up to date to make them more
> > > > > useful to more people.
> > > > >
> > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <thenatog@gmail.com
> > > > <mailto:
> > > > > thenatog@gmail.com>> wrote:
> > > > > I'm a +1 for removing pretty much all of this stuff. There are
> > security
> > > > > implications to keeping old dependencies around, so the more old code
> > > we
> > > > > can remove the better. I agree that eventually we need to move to
> > > > > supporting only Java 11+, and as our next release will probably be
> > > about
> > > > 4
> > > > > - 6 months from now that doesn't seem too soon. We could potentially
> > > > break
> > > > > this in two and remove the deprecated processors and leave 1.x on
> > Java
> > > 8,
> > > > > and finally start on 2.x which would support only Java 11. I'm unsure
> > > of
> > > > > what implications changing the date and time handling would have -
> > for
> > > > > running systems that use long term historical logs, unexpected
> > impacts
> > > to
> > > > > time logging could be a problem.
> > > > >
> > > > > As Joe says I think feature work will have to be dedicated to 2.x and
> > > we
> > > > > could support 1.x for security fixes for some period of time. 2.x
> > seems
> > > > > like a gargantuan task but it's probably time to get started. Not
> > sure
> > > > how
> > > > > we handle all open PRs and the transition between 1.x and 2.x.
> > > > >
> > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <joe.witt@gmail.com
> > <mailto:
> > > > > joe.witt@gmail.com>> wrote:
> > > > >
> > > > > Jon
> > > > >
> > > > > You're right we have to be careful and you're right there are still
> > > > > significant Java 8 users out there. But we also have to be careful
> > > > > about security and sustainability of the codebase. If we had talked
> > > > > about this last year when that article came out I'd have agreed it is
> > > > > too early. Interestingly that link seems to get updated and I tried
> > > > > [1] and found more recent data (not sure how recent). Anyway it
> > > > > suggests Java 8 is still the top dog but we see good growth on 11. In
> > > > > my $dayjob this aligns to what I'm seeing too. Customers didn't seem
> > > > > to care about Java 11 until later half last year and now suddenly it
> > > > > is all over the place.
> > > > >
> > > > > I think once we put out a NiFi 2.0 release we'd see rapid decrease in
> > > > > work on the 1.x line just being blunt. We did this many years ago
> > > > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
> > > > > so) but it was purely bug fix/security related bits. We would need to
> > > > > do something similar. But feature work would almost certainly go to
> > > > > the 2.x line. Maybe there are other workable models but my instinct
> > > > > suggests this is likely to follow a similar path.
> > > > >
> > > > > ...anyway I agree it isn't that easy of a call to dump Java 8. We
> > > > > need to make the call in both the interests of the user base and the
> > > > > contributor base of the community.
> > > > >
> > > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > > >
> > > > >
> > > > > Thanks
> > > > > Joe
> > > > >
> > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <joe.witt@gmail.com<mailto:
> > > > > joe.witt@gmail.com>> wrote:
> > > > > Russ
> > > > >
> > > > > Yeah the flow registry is a key part of it. But also now you can
> > > > > download the flow definition in JSON (upload i think is there now
> > > > > too). Templates offered a series of challenges such as we store them
> > > > > in the flow definition which has made flows massive in an unintended
> > > > > way which isn't fun for cluster behavior.
> > > > >
> > > > > We have a couple cases where we headed down a particular concept and
> > > > > came up with better approaches later. We need to reconcile these with
> > > > > the benefit of hindsight, and while being careful to be not overly
> > > > > disruptive to existing users, to reduce the codebase/maintenance
> > > > > burden and allow continued evolution of the project.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > russ@windofkeltia.com
> > > > > <ma...@windofkeltia.com>>
> > > > > wrote:
> > > > > Joe,
> > > > >
> > > > > I apologize for the off-topic intrusion, but what replaces templates?
> > > > > The Registry? Templates rocked and we have used them since 0.5.x.
> > > > >
> > > > > Russ
> > > > >
> > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > David,
> > > > >
> > > > > I think this is a highly reasonable approach and such a focus will
> > > > > greatly help make a 2.0 release far more approachable to knock out.
> > > > > Not only that but tech debt reduction would help make work towards
> > > > > major features we'd think about in a 'major release' sense more
> > > > > approachable.
> > > > >
> > > > > We should remove all deprecated things (as well as verify we have the
> > > > > right list). We should remove/consider removal of deprecated
> > > > > concepts
> > > > > like templates. We should consider whether we can resolve the
> > > > > various
> > > > > ways we've handled what are now parameters down to one clean
> > > > > approach.
> > > > > We should remove options in the nifi.properties which turn out to
> > > > > never be used quite right (if there are). There is quite a bit we
> > > > > can
> > > > > do purely in the name of tech debt reduction.
> > > > >
> > > > > Lots to consider here but I think this is the right discussion.
> > > > >
> > > > > Than ks
> > > > >
> > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bbende@gmail.com
> > <mailto:
> > > > > bbende@gmail.com>>
> > > > > wrote:
> > > > > I'm a +1 for this... Not sure if this falls under "Removing
> > > > > Deprecated
> > > > > Components", but I think we should also look at anything that has
> > > > > been
> > > > > marked as deprecated throughout the code base as a candidate for
> > > > > removal. There are quite a few classes, methods, properties, etc
> > > > > that
> > > > > have been waiting for a chance to be removed.
> > > > >
> > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > > <ex...@apache.org>>
> > > wrote:
> > > > > Team,
> > > > >
> > > > > With all of the excellent work that many have contributed to NiFi
> > > > > over the
> > > > > years, the code base has also accumulated some amount of technical
> > > > > debt. A
> > > > > handful of components have been marked as deprecated, and some
> > > > > components
> > > > > remain in the code base to support integration with old versions
> > > > > of various
> > > > > products. Following the principles of semantic versioning,
> > > > > introducing a
> > > > > major release would provide the opportunity to remove these
> > > > > deprecated and
> > > > > unsupported components.
> > > > >
> > > > > Rather than focusing the next major release on new features, what
> > > > > do you
> > > > > think about focusing on technical debt removal? This approach
> > > > > would not
> > > > > make for the most interesting release, but it provides the
> > > > > opportunity to
> > > > > clean up elements that involve breaking changes.
> > > > >
> > > > > Focusing on technical debt, at least three primary goals come to
> > > > > mind for
> > > > > the next major release:
> > > > >
> > > > > 1. Removal of deprecated and unmaintained components
> > > > > 2. Require Java 11 as the minimum supported version
> > > > > 3. Transition internal date and time handling to JSR 310 java.time
> > > > > components
> > > > >
> > > > > *Removing Deprecated Components*
> > > > >
> > > > > Removing support for older and deprecated components provides a
> > > > > great
> > > > > opportunity to improve the overall security posture when it comes
> > > > > to
> > > > > maintaining dependencies. The OWASP dependency plugin report
> > > > > currently
> > > > > generates 50 MB of HTML for questionable dependencies, many of
> > > > > which are
> > > > > related to old versions of various libraries.
> > > > >
> > > > > As a starting point, here are a handful of components and
> > > > > extension modules
> > > > > that could be targeted for removal in a major version:
> > > > >
> > > > > - PostHTTP and GetHTTP
> > > > > - ListenLumberjack and the entire nifi-lumberjack-bundle
> > > > > - ListenBeats and the entire nifi-beats-bundle
> > > > > - Elasticsearch 5 components
> > > > > - Hive 1 and 2 components
> > > > >
> > > > > *Requiring Java 11*
> > > > >
> > > > > Java 8 is now over seven years old, and NiFi has supported general
> > > > > compatibility with Java 11 for several years. NiFi 1.14.0
> > > > > incorporated
> > > > > internal improvements specifically related to TLS 1.3, which
> > > > > allowed
> > > > > closing out the long-running Java 11 compatibility epic NIFI-5174.
> > > > > Making
> > > > > Java 11 the minimum required version provides the opportunity to
> > > > > address
> > > > > any lingering edge cases and put NiFi in a better position to
> > > > > support
> > > > > current Java versions.
> > > > >
> > > > > *JSR 310 for Date and Time Handling*
> > > > >
> > > > > Without making the scope too broad, transitioning internal date
> > > > > and time
> > > > > handling to use DateTimeFormatter instead of SimpleDateFormat
> > > > > would provide
> > > > > a number of advantages. The Java Time components provide much
> > > > > better
> > > > > clarity when it comes to handling localized date and time
> > > > > representations,
> > > > > and also avoid the inherent confusion of java.sql.Date extending
> > > > > java.util.Date. Many internal components, specifically
> > > > > Record-oriented
> > > > > processors and services, rely on date parsing, leading to
> > > > > confusion and
> > > > > various workarounds. The pattern formats of SimpleDateFormat and
> > > > > DateTimeFormatter are very similar, but there are a few subtle
> > > > > differences.
> > > > > Making this transition would provide a much better foundation
> > > > > going forward.
> > > > > *Conclusion*
> > > > >
> > > > > Thanks for giving this proposal some consideration. Many of you
> > > > > have been
> > > > > developing NiFi for years and I look forward to your feedback. I
> > > > > would be
> > > > > glad to put together a more formalized recommendation on
> > > > > Confluence and
> > > > > write up Jira epics if this general approach sounds agreeable to
> > > > > the
> > > > > community.
> > > > >
> > > > > Regards,
> > > > > David Handermann
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Edward Armes <ed...@gmail.com>.
Given the major version number shift and the spliting up of processors into
multiple NAR's. I'd like to suggest that we start individually versioning
individual NARs/ bundles.
I can see this bringing a large number of benifits including making Nifi
more deployable with things RPM, but also potentially allowing those that
have to deploy managed Nifi instances easier to upgrade.
Edward
On Mon, 26 Jul 2021, 20:42 Otto Fowler, <ot...@gmail.com> wrote:
> The issue with updating the aws sdk is if it breaks any one of the
> processors.
> the Web Gateway API invoke processor for example is not using a high level
> purpose build client and may break.
>
> If we change the aws version, we need to coordinate in such a way that they
> all
> can come along reasonably.
> IE: what happens if 1 or 2 break but the rest or OK?
>
>
>
> From: David Handermann <ex...@apache.org>
> <ex...@apache.org>
> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> Date: July 26, 2021 at 09:33:42
> To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
>
> Chris,
>
> Thanks for the reply and recommendations. It seems like some of the work to
> reorganize the module structure could be done outside of a major release,
> but it would be great to target any breaking changes for 2.0. Perhaps a
> separate feature proposal on module restructuring, with the goal of
> supporting optimized builds, would be a helpful way to move that part of
> the discussion forward.
>
> Regarding updating AWS SDK to version 2, it seems like that might be
> possible now. I haven't taken a close look at the referencing components,
> so I'm not sure about the level of effort involved. Minor NiFi version
> updates have incorporated major new versions of dependencies. For example,
> NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the one
> hand, including the AWS SDK update as part of a major release seems
> helpful, but unless there are changes that break existing component
> properties, upgrading the AWS SDK could be worked independently. Others may
> have more insight into particular usage of that library.
>
> Regards,
> David Handermann
>
> On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> <ch...@naimuri.com.invalid> wrote:
>
> > Might be worth considering refactoring the build as part of this work
> too,
> > e.g. only building the bits of the repo affected by a commit, etc. -
> > discussed briefly in previous threads but don't think any changes made
> yet.
> > If NARs/components are likely to be split up and refactored then such
> work
> > around the build probably makes sense to consider.
> >
> > I've a couple of PRs open that include updates to Elasticsearch versions
> > already, although I stopped at 7.10.2 (after which Elastic changed
> licence)
> > in case there were licence concerns. But more work can be done to tidy up
> > the processors, absolutely.
> >
> > AWS libraries to v2 would seem a sensible move and a refactor of those
> > processors as well.
> >
> >
> > Cheers,
> >
> > Chris Sampson
> >
> > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> exceptionfactory@apache.org>
>
> > wrote:
> >
> > > Thanks for pointing out the standard NAR bundles, Mark. There are a
> > number
> > > of components in the standard NAR bundles with particular dependencies
> > that
> > > would make more sense in separate NARs. Reorganizing the standard NAR
> to
> > > components with limited dependencies and wide applicability would
> > > definitely help with future maintenance.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <ma...@hotmail.com>
> > wrote:
> > >
> > > > There’s also some code that exists in order to maintain backward
> > > > compatibility in the repositories. I would very much like the
> > > repositories
> > > > to contain no unnecessary code. And swap file format supports really
> > old
> > > > formats. And the old impls of the repositories themselves, like
> > > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff
> > > there
> > > > that can be removed. And some methods in ProcessSession that are
> never
> > > used
> > > > by any processor in the codebase but exists in the public API so
> can’t
> > be
> > > > removed till 2.0.
> > > >
> > > > I think his is also a great time to clean up the “standard nar.” At
> > this
> > > > point, it’s something like 70 MB. And many of the components there
> are
> > > not
> > > > really “standard” - things like connecting to FTP & SFTP servers, XML
> > > > processing, Jolt transform, etc. could potentially be moved into
> other
> > > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is 6.9 MB
> is
> > > not
> > > > necessary for stateless or minifi java. Lots of things probably to
> > > > reconsider within the standard nar.
> > > >
> > > > I definitely think this is a reasonable approach, to allow for a 2.0
> > that
> > > > is not a huge feature release but allows the project to be simpler
> and
> > > more
> > > > nimble.
> > > >
> > > > Thanks
> > > > -Mark
> > > >
> > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <mikerthomsen@gmail.com
> > > <mailto:
> > > > mikerthomsen@gmail.com>> wrote:
> > > >
> > > > Russell,
> > > >
> > > > AFAICT from looking at Elastic's repos, the low level REST client is
> > > > still fine.
> > > >
> > >
> >
>
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > > >
> > > > Our Elasticsearch support is spread over two NARs at present. One
> uses
> > > > OkHttp and the other uses that low level Elastic REST client.
> > > > Therefore, I think we're fine on licensing for the moment.
> > > >
> > > > Mike
> > > >
> > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <
> russ@windofkeltia.com
> > > > <ma...@windofkeltia.com>> wrote:
> > > >
> > > > Bringing up Elastic also reminds me that the Elastic framework has
> just
> > > > recently transitioned out of Open Source, so to acknowledge that,
> maybe
> > > > some effort toward OpenSearch--I say this not understanding exactly
> how
> > > > this sort of thing is considered in a large-scale, world-class
> software
> > > > project like Apache NiFi. (I'm not a contributor, just a grateful
> > > > consumer.)
> > > >
> > > > Russ
> > > >
> > > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > > Along with the itemized list for ancient components we should look at
> > > > updating versions of drivers, SDKs, etc. for external systems such as
> > > > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0
> > > > is probably the right time to get things up to date to make them more
> > > > useful to more people.
> > > >
> > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <thenatog@gmail.com
> > > <mailto:
> > > > thenatog@gmail.com>> wrote:
> > > > I'm a +1 for removing pretty much all of this stuff. There are
> security
> > > > implications to keeping old dependencies around, so the more old code
> > we
> > > > can remove the better. I agree that eventually we need to move to
> > > > supporting only Java 11+, and as our next release will probably be
> > about
> > > 4
> > > > - 6 months from now that doesn't seem too soon. We could potentially
> > > break
> > > > this in two and remove the deprecated processors and leave 1.x on
> Java
> > 8,
> > > > and finally start on 2.x which would support only Java 11. I'm unsure
> > of
> > > > what implications changing the date and time handling would have -
> for
> > > > running systems that use long term historical logs, unexpected
> impacts
> > to
> > > > time logging could be a problem.
> > > >
> > > > As Joe says I think feature work will have to be dedicated to 2.x and
> > we
> > > > could support 1.x for security fixes for some period of time. 2.x
> seems
> > > > like a gargantuan task but it's probably time to get started. Not
> sure
> > > how
> > > > we handle all open PRs and the transition between 1.x and 2.x.
> > > >
> > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <joe.witt@gmail.com
> <mailto:
> > > > joe.witt@gmail.com>> wrote:
> > > >
> > > > Jon
> > > >
> > > > You're right we have to be careful and you're right there are still
> > > > significant Java 8 users out there. But we also have to be careful
> > > > about security and sustainability of the codebase. If we had talked
> > > > about this last year when that article came out I'd have agreed it is
> > > > too early. Interestingly that link seems to get updated and I tried
> > > > [1] and found more recent data (not sure how recent). Anyway it
> > > > suggests Java 8 is still the top dog but we see good growth on 11. In
> > > > my $dayjob this aligns to what I'm seeing too. Customers didn't seem
> > > > to care about Java 11 until later half last year and now suddenly it
> > > > is all over the place.
> > > >
> > > > I think once we put out a NiFi 2.0 release we'd see rapid decrease in
> > > > work on the 1.x line just being blunt. We did this many years ago
> > > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
> > > > so) but it was purely bug fix/security related bits. We would need to
> > > > do something similar. But feature work would almost certainly go to
> > > > the 2.x line. Maybe there are other workable models but my instinct
> > > > suggests this is likely to follow a similar path.
> > > >
> > > > ...anyway I agree it isn't that easy of a call to dump Java 8. We
> > > > need to make the call in both the interests of the user base and the
> > > > contributor base of the community.
> > > >
> > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > >
> > > >
> > > > Thanks
> > > > Joe
> > > >
> > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <joe.witt@gmail.com<mailto:
> > > > joe.witt@gmail.com>> wrote:
> > > > Russ
> > > >
> > > > Yeah the flow registry is a key part of it. But also now you can
> > > > download the flow definition in JSON (upload i think is there now
> > > > too). Templates offered a series of challenges such as we store them
> > > > in the flow definition which has made flows massive in an unintended
> > > > way which isn't fun for cluster behavior.
> > > >
> > > > We have a couple cases where we headed down a particular concept and
> > > > came up with better approaches later. We need to reconcile these with
> > > > the benefit of hindsight, and while being careful to be not overly
> > > > disruptive to existing users, to reduce the codebase/maintenance
> > > > burden and allow continued evolution of the project.
> > > >
> > > > Thanks
> > > >
> > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> russ@windofkeltia.com
> > > > <ma...@windofkeltia.com>>
> > > > wrote:
> > > > Joe,
> > > >
> > > > I apologize for the off-topic intrusion, but what replaces templates?
> > > > The Registry? Templates rocked and we have used them since 0.5.x.
> > > >
> > > > Russ
> > > >
> > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > David,
> > > >
> > > > I think this is a highly reasonable approach and such a focus will
> > > > greatly help make a 2.0 release far more approachable to knock out.
> > > > Not only that but tech debt reduction would help make work towards
> > > > major features we'd think about in a 'major release' sense more
> > > > approachable.
> > > >
> > > > We should remove all deprecated things (as well as verify we have the
> > > > right list). We should remove/consider removal of deprecated
> > > > concepts
> > > > like templates. We should consider whether we can resolve the
> > > > various
> > > > ways we've handled what are now parameters down to one clean
> > > > approach.
> > > > We should remove options in the nifi.properties which turn out to
> > > > never be used quite right (if there are). There is quite a bit we
> > > > can
> > > > do purely in the name of tech debt reduction.
> > > >
> > > > Lots to consider here but I think this is the right discussion.
> > > >
> > > > Than ks
> > > >
> > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bbende@gmail.com
> <mailto:
> > > > bbende@gmail.com>>
> > > > wrote:
> > > > I'm a +1 for this... Not sure if this falls under "Removing
> > > > Deprecated
> > > > Components", but I think we should also look at anything that has
> > > > been
> > > > marked as deprecated throughout the code base as a candidate for
> > > > removal. There are quite a few classes, methods, properties, etc
> > > > that
> > > > have been waiting for a chance to be removed.
> > > >
> > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > <ex...@apache.org>>
> > wrote:
> > > > Team,
> > > >
> > > > With all of the excellent work that many have contributed to NiFi
> > > > over the
> > > > years, the code base has also accumulated some amount of technical
> > > > debt. A
> > > > handful of components have been marked as deprecated, and some
> > > > components
> > > > remain in the code base to support integration with old versions
> > > > of various
> > > > products. Following the principles of semantic versioning,
> > > > introducing a
> > > > major release would provide the opportunity to remove these
> > > > deprecated and
> > > > unsupported components.
> > > >
> > > > Rather than focusing the next major release on new features, what
> > > > do you
> > > > think about focusing on technical debt removal? This approach
> > > > would not
> > > > make for the most interesting release, but it provides the
> > > > opportunity to
> > > > clean up elements that involve breaking changes.
> > > >
> > > > Focusing on technical debt, at least three primary goals come to
> > > > mind for
> > > > the next major release:
> > > >
> > > > 1. Removal of deprecated and unmaintained components
> > > > 2. Require Java 11 as the minimum supported version
> > > > 3. Transition internal date and time handling to JSR 310 java.time
> > > > components
> > > >
> > > > *Removing Deprecated Components*
> > > >
> > > > Removing support for older and deprecated components provides a
> > > > great
> > > > opportunity to improve the overall security posture when it comes
> > > > to
> > > > maintaining dependencies. The OWASP dependency plugin report
> > > > currently
> > > > generates 50 MB of HTML for questionable dependencies, many of
> > > > which are
> > > > related to old versions of various libraries.
> > > >
> > > > As a starting point, here are a handful of components and
> > > > extension modules
> > > > that could be targeted for removal in a major version:
> > > >
> > > > - PostHTTP and GetHTTP
> > > > - ListenLumberjack and the entire nifi-lumberjack-bundle
> > > > - ListenBeats and the entire nifi-beats-bundle
> > > > - Elasticsearch 5 components
> > > > - Hive 1 and 2 components
> > > >
> > > > *Requiring Java 11*
> > > >
> > > > Java 8 is now over seven years old, and NiFi has supported general
> > > > compatibility with Java 11 for several years. NiFi 1.14.0
> > > > incorporated
> > > > internal improvements specifically related to TLS 1.3, which
> > > > allowed
> > > > closing out the long-running Java 11 compatibility epic NIFI-5174.
> > > > Making
> > > > Java 11 the minimum required version provides the opportunity to
> > > > address
> > > > any lingering edge cases and put NiFi in a better position to
> > > > support
> > > > current Java versions.
> > > >
> > > > *JSR 310 for Date and Time Handling*
> > > >
> > > > Without making the scope too broad, transitioning internal date
> > > > and time
> > > > handling to use DateTimeFormatter instead of SimpleDateFormat
> > > > would provide
> > > > a number of advantages. The Java Time components provide much
> > > > better
> > > > clarity when it comes to handling localized date and time
> > > > representations,
> > > > and also avoid the inherent confusion of java.sql.Date extending
> > > > java.util.Date. Many internal components, specifically
> > > > Record-oriented
> > > > processors and services, rely on date parsing, leading to
> > > > confusion and
> > > > various workarounds. The pattern formats of SimpleDateFormat and
> > > > DateTimeFormatter are very similar, but there are a few subtle
> > > > differences.
> > > > Making this transition would provide a much better foundation
> > > > going forward.
> > > > *Conclusion*
> > > >
> > > > Thanks for giving this proposal some consideration. Many of you
> > > > have been
> > > > developing NiFi for years and I look forward to your feedback. I
> > > > would be
> > > > glad to put together a more formalized recommendation on
> > > > Confluence and
> > > > write up Jira epics if this general approach sounds agreeable to
> > > > the
> > > > community.
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Otto Fowler <ot...@gmail.com>.
The issue with updating the aws sdk is if it breaks any one of the
processors.
the Web Gateway API invoke processor for example is not using a high level
purpose build client and may break.
If we change the aws version, we need to coordinate in such a way that they
all
can come along reasonably.
IE: what happens if 1 or 2 break but the rest or OK?
From: David Handermann <ex...@apache.org>
<ex...@apache.org>
Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
Date: July 26, 2021 at 09:33:42
To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
Chris,
Thanks for the reply and recommendations. It seems like some of the work to
reorganize the module structure could be done outside of a major release,
but it would be great to target any breaking changes for 2.0. Perhaps a
separate feature proposal on module restructuring, with the goal of
supporting optimized builds, would be a helpful way to move that part of
the discussion forward.
Regarding updating AWS SDK to version 2, it seems like that might be
possible now. I haven't taken a close look at the referencing components,
so I'm not sure about the level of effort involved. Minor NiFi version
updates have incorporated major new versions of dependencies. For example,
NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the one
hand, including the AWS SDK update as part of a major release seems
helpful, but unless there are changes that break existing component
properties, upgrading the AWS SDK could be worked independently. Others may
have more insight into particular usage of that library.
Regards,
David Handermann
On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
<ch...@naimuri.com.invalid> wrote:
> Might be worth considering refactoring the build as part of this work
too,
> e.g. only building the bits of the repo affected by a commit, etc. -
> discussed briefly in previous threads but don't think any changes made
yet.
> If NARs/components are likely to be split up and refactored then such
work
> around the build probably makes sense to consider.
>
> I've a couple of PRs open that include updates to Elasticsearch versions
> already, although I stopped at 7.10.2 (after which Elastic changed
licence)
> in case there were licence concerns. But more work can be done to tidy up
> the processors, absolutely.
>
> AWS libraries to v2 would seem a sensible move and a refactor of those
> processors as well.
>
>
> Cheers,
>
> Chris Sampson
>
> On Sat, 24 Jul 2021, 17:47 David Handermann, <ex...@apache.org>
> wrote:
>
> > Thanks for pointing out the standard NAR bundles, Mark. There are a
> number
> > of components in the standard NAR bundles with particular dependencies
> that
> > would make more sense in separate NARs. Reorganizing the standard NAR
to
> > components with limited dependencies and wide applicability would
> > definitely help with future maintenance.
> >
> > Regards,
> > David Handermann
> >
> > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <ma...@hotmail.com>
> wrote:
> >
> > > There’s also some code that exists in order to maintain backward
> > > compatibility in the repositories. I would very much like the
> > repositories
> > > to contain no unnecessary code. And swap file format supports really
> old
> > > formats. And the old impls of the repositories themselves, like
> > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff
> > there
> > > that can be removed. And some methods in ProcessSession that are
never
> > used
> > > by any processor in the codebase but exists in the public API so
can’t
> be
> > > removed till 2.0.
> > >
> > > I think his is also a great time to clean up the “standard nar.” At
> this
> > > point, it’s something like 70 MB. And many of the components there
are
> > not
> > > really “standard” - things like connecting to FTP & SFTP servers, XML
> > > processing, Jolt transform, etc. could potentially be moved into
other
> > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is 6.9 MB
is
> > not
> > > necessary for stateless or minifi java. Lots of things probably to
> > > reconsider within the standard nar.
> > >
> > > I definitely think this is a reasonable approach, to allow for a 2.0
> that
> > > is not a huge feature release but allows the project to be simpler
and
> > more
> > > nimble.
> > >
> > > Thanks
> > > -Mark
> > >
> > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <mikerthomsen@gmail.com
> > <mailto:
> > > mikerthomsen@gmail.com>> wrote:
> > >
> > > Russell,
> > >
> > > AFAICT from looking at Elastic's repos, the low level REST client is
> > > still fine.
> > >
> >
>
https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > >
> > > Our Elasticsearch support is spread over two NARs at present. One
uses
> > > OkHttp and the other uses that low level Elastic REST client.
> > > Therefore, I think we're fine on licensing for the moment.
> > >
> > > Mike
> > >
> > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <russ@windofkeltia.com
> > > <ma...@windofkeltia.com>> wrote:
> > >
> > > Bringing up Elastic also reminds me that the Elastic framework has
just
> > > recently transitioned out of Open Source, so to acknowledge that,
maybe
> > > some effort toward OpenSearch--I say this not understanding exactly
how
> > > this sort of thing is considered in a large-scale, world-class
software
> > > project like Apache NiFi. (I'm not a contributor, just a grateful
> > > consumer.)
> > >
> > > Russ
> > >
> > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > Along with the itemized list for ancient components we should look at
> > > updating versions of drivers, SDKs, etc. for external systems such as
> > > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0
> > > is probably the right time to get things up to date to make them more
> > > useful to more people.
> > >
> > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <thenatog@gmail.com
> > <mailto:
> > > thenatog@gmail.com>> wrote:
> > > I'm a +1 for removing pretty much all of this stuff. There are
security
> > > implications to keeping old dependencies around, so the more old code
> we
> > > can remove the better. I agree that eventually we need to move to
> > > supporting only Java 11+, and as our next release will probably be
> about
> > 4
> > > - 6 months from now that doesn't seem too soon. We could potentially
> > break
> > > this in two and remove the deprecated processors and leave 1.x on
Java
> 8,
> > > and finally start on 2.x which would support only Java 11. I'm unsure
> of
> > > what implications changing the date and time handling would have -
for
> > > running systems that use long term historical logs, unexpected
impacts
> to
> > > time logging could be a problem.
> > >
> > > As Joe says I think feature work will have to be dedicated to 2.x and
> we
> > > could support 1.x for security fixes for some period of time. 2.x
seems
> > > like a gargantuan task but it's probably time to get started. Not
sure
> > how
> > > we handle all open PRs and the transition between 1.x and 2.x.
> > >
> > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <joe.witt@gmail.com<mailto:
> > > joe.witt@gmail.com>> wrote:
> > >
> > > Jon
> > >
> > > You're right we have to be careful and you're right there are still
> > > significant Java 8 users out there. But we also have to be careful
> > > about security and sustainability of the codebase. If we had talked
> > > about this last year when that article came out I'd have agreed it is
> > > too early. Interestingly that link seems to get updated and I tried
> > > [1] and found more recent data (not sure how recent). Anyway it
> > > suggests Java 8 is still the top dog but we see good growth on 11. In
> > > my $dayjob this aligns to what I'm seeing too. Customers didn't seem
> > > to care about Java 11 until later half last year and now suddenly it
> > > is all over the place.
> > >
> > > I think once we put out a NiFi 2.0 release we'd see rapid decrease in
> > > work on the 1.x line just being blunt. We did this many years ago
> > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
> > > so) but it was purely bug fix/security related bits. We would need to
> > > do something similar. But feature work would almost certainly go to
> > > the 2.x line. Maybe there are other workable models but my instinct
> > > suggests this is likely to follow a similar path.
> > >
> > > ...anyway I agree it isn't that easy of a call to dump Java 8. We
> > > need to make the call in both the interests of the user base and the
> > > contributor base of the community.
> > >
> > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> > >
> > >
> > > Thanks
> > > Joe
> > >
> > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <joe.witt@gmail.com<mailto:
> > > joe.witt@gmail.com>> wrote:
> > > Russ
> > >
> > > Yeah the flow registry is a key part of it. But also now you can
> > > download the flow definition in JSON (upload i think is there now
> > > too). Templates offered a series of challenges such as we store them
> > > in the flow definition which has made flows massive in an unintended
> > > way which isn't fun for cluster behavior.
> > >
> > > We have a couple cases where we headed down a particular concept and
> > > came up with better approaches later. We need to reconcile these with
> > > the benefit of hindsight, and while being careful to be not overly
> > > disruptive to existing users, to reduce the codebase/maintenance
> > > burden and allow continued evolution of the project.
> > >
> > > Thanks
> > >
> > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <russ@windofkeltia.com
> > > <ma...@windofkeltia.com>>
> > > wrote:
> > > Joe,
> > >
> > > I apologize for the off-topic intrusion, but what replaces templates?
> > > The Registry? Templates rocked and we have used them since 0.5.x.
> > >
> > > Russ
> > >
> > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > David,
> > >
> > > I think this is a highly reasonable approach and such a focus will
> > > greatly help make a 2.0 release far more approachable to knock out.
> > > Not only that but tech debt reduction would help make work towards
> > > major features we'd think about in a 'major release' sense more
> > > approachable.
> > >
> > > We should remove all deprecated things (as well as verify we have the
> > > right list). We should remove/consider removal of deprecated
> > > concepts
> > > like templates. We should consider whether we can resolve the
> > > various
> > > ways we've handled what are now parameters down to one clean
> > > approach.
> > > We should remove options in the nifi.properties which turn out to
> > > never be used quite right (if there are). There is quite a bit we
> > > can
> > > do purely in the name of tech debt reduction.
> > >
> > > Lots to consider here but I think this is the right discussion.
> > >
> > > Than ks
> > >
> > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bbende@gmail.com<mailto:
> > > bbende@gmail.com>>
> > > wrote:
> > > I'm a +1 for this... Not sure if this falls under "Removing
> > > Deprecated
> > > Components", but I think we should also look at anything that has
> > > been
> > > marked as deprecated throughout the code base as a candidate for
> > > removal. There are quite a few classes, methods, properties, etc
> > > that
> > > have been waiting for a chance to be removed.
> > >
> > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > <ex...@apache.org>>
> wrote:
> > > Team,
> > >
> > > With all of the excellent work that many have contributed to NiFi
> > > over the
> > > years, the code base has also accumulated some amount of technical
> > > debt. A
> > > handful of components have been marked as deprecated, and some
> > > components
> > > remain in the code base to support integration with old versions
> > > of various
> > > products. Following the principles of semantic versioning,
> > > introducing a
> > > major release would provide the opportunity to remove these
> > > deprecated and
> > > unsupported components.
> > >
> > > Rather than focusing the next major release on new features, what
> > > do you
> > > think about focusing on technical debt removal? This approach
> > > would not
> > > make for the most interesting release, but it provides the
> > > opportunity to
> > > clean up elements that involve breaking changes.
> > >
> > > Focusing on technical debt, at least three primary goals come to
> > > mind for
> > > the next major release:
> > >
> > > 1. Removal of deprecated and unmaintained components
> > > 2. Require Java 11 as the minimum supported version
> > > 3. Transition internal date and time handling to JSR 310 java.time
> > > components
> > >
> > > *Removing Deprecated Components*
> > >
> > > Removing support for older and deprecated components provides a
> > > great
> > > opportunity to improve the overall security posture when it comes
> > > to
> > > maintaining dependencies. The OWASP dependency plugin report
> > > currently
> > > generates 50 MB of HTML for questionable dependencies, many of
> > > which are
> > > related to old versions of various libraries.
> > >
> > > As a starting point, here are a handful of components and
> > > extension modules
> > > that could be targeted for removal in a major version:
> > >
> > > - PostHTTP and GetHTTP
> > > - ListenLumberjack and the entire nifi-lumberjack-bundle
> > > - ListenBeats and the entire nifi-beats-bundle
> > > - Elasticsearch 5 components
> > > - Hive 1 and 2 components
> > >
> > > *Requiring Java 11*
> > >
> > > Java 8 is now over seven years old, and NiFi has supported general
> > > compatibility with Java 11 for several years. NiFi 1.14.0
> > > incorporated
> > > internal improvements specifically related to TLS 1.3, which
> > > allowed
> > > closing out the long-running Java 11 compatibility epic NIFI-5174.
> > > Making
> > > Java 11 the minimum required version provides the opportunity to
> > > address
> > > any lingering edge cases and put NiFi in a better position to
> > > support
> > > current Java versions.
> > >
> > > *JSR 310 for Date and Time Handling*
> > >
> > > Without making the scope too broad, transitioning internal date
> > > and time
> > > handling to use DateTimeFormatter instead of SimpleDateFormat
> > > would provide
> > > a number of advantages. The Java Time components provide much
> > > better
> > > clarity when it comes to handling localized date and time
> > > representations,
> > > and also avoid the inherent confusion of java.sql.Date extending
> > > java.util.Date. Many internal components, specifically
> > > Record-oriented
> > > processors and services, rely on date parsing, leading to
> > > confusion and
> > > various workarounds. The pattern formats of SimpleDateFormat and
> > > DateTimeFormatter are very similar, but there are a few subtle
> > > differences.
> > > Making this transition would provide a much better foundation
> > > going forward.
> > > *Conclusion*
> > >
> > > Thanks for giving this proposal some consideration. Many of you
> > > have been
> > > developing NiFi for years and I look forward to your feedback. I
> > > would be
> > > glad to put together a more formalized recommendation on
> > > Confluence and
> > > write up Jira epics if this general approach sounds agreeable to
> > > the
> > > community.
> > >
> > > Regards,
> > > David Handermann
> > >
> > >
> > >
> >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by David Handermann <ex...@apache.org>.
Chris,
Thanks for the reply and recommendations. It seems like some of the work to
reorganize the module structure could be done outside of a major release,
but it would be great to target any breaking changes for 2.0. Perhaps a
separate feature proposal on module restructuring, with the goal of
supporting optimized builds, would be a helpful way to move that part of
the discussion forward.
Regarding updating AWS SDK to version 2, it seems like that might be
possible now. I haven't taken a close look at the referencing components,
so I'm not sure about the level of effort involved. Minor NiFi version
updates have incorporated major new versions of dependencies. For example,
NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the one
hand, including the AWS SDK update as part of a major release seems
helpful, but unless there are changes that break existing component
properties, upgrading the AWS SDK could be worked independently. Others may
have more insight into particular usage of that library.
Regards,
David Handermann
On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
<ch...@naimuri.com.invalid> wrote:
> Might be worth considering refactoring the build as part of this work too,
> e.g. only building the bits of the repo affected by a commit, etc. -
> discussed briefly in previous threads but don't think any changes made yet.
> If NARs/components are likely to be split up and refactored then such work
> around the build probably makes sense to consider.
>
> I've a couple of PRs open that include updates to Elasticsearch versions
> already, although I stopped at 7.10.2 (after which Elastic changed licence)
> in case there were licence concerns. But more work can be done to tidy up
> the processors, absolutely.
>
> AWS libraries to v2 would seem a sensible move and a refactor of those
> processors as well.
>
>
> Cheers,
>
> Chris Sampson
>
> On Sat, 24 Jul 2021, 17:47 David Handermann, <ex...@apache.org>
> wrote:
>
> > Thanks for pointing out the standard NAR bundles, Mark. There are a
> number
> > of components in the standard NAR bundles with particular dependencies
> that
> > would make more sense in separate NARs. Reorganizing the standard NAR to
> > components with limited dependencies and wide applicability would
> > definitely help with future maintenance.
> >
> > Regards,
> > David Handermann
> >
> > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <ma...@hotmail.com>
> wrote:
> >
> > > There’s also some code that exists in order to maintain backward
> > > compatibility in the repositories. I would very much like the
> > repositories
> > > to contain no unnecessary code. And swap file format supports really
> old
> > > formats. And the old impls of the repositories themselves, like
> > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff
> > there
> > > that can be removed. And some methods in ProcessSession that are never
> > used
> > > by any processor in the codebase but exists in the public API so can’t
> be
> > > removed till 2.0.
> > >
> > > I think his is also a great time to clean up the “standard nar.” At
> this
> > > point, it’s something like 70 MB. And many of the components there are
> > not
> > > really “standard” - things like connecting to FTP & SFTP servers, XML
> > > processing, Jolt transform, etc. could potentially be moved into other
> > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is 6.9 MB is
> > not
> > > necessary for stateless or minifi java. Lots of things probably to
> > > reconsider within the standard nar.
> > >
> > > I definitely think this is a reasonable approach, to allow for a 2.0
> that
> > > is not a huge feature release but allows the project to be simpler and
> > more
> > > nimble.
> > >
> > > Thanks
> > > -Mark
> > >
> > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <mikerthomsen@gmail.com
> > <mailto:
> > > mikerthomsen@gmail.com>> wrote:
> > >
> > > Russell,
> > >
> > > AFAICT from looking at Elastic's repos, the low level REST client is
> > > still fine.
> > >
> >
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > >
> > > Our Elasticsearch support is spread over two NARs at present. One uses
> > > OkHttp and the other uses that low level Elastic REST client.
> > > Therefore, I think we're fine on licensing for the moment.
> > >
> > > Mike
> > >
> > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <russ@windofkeltia.com
> > > <ma...@windofkeltia.com>> wrote:
> > >
> > > Bringing up Elastic also reminds me that the Elastic framework has just
> > > recently transitioned out of Open Source, so to acknowledge that, maybe
> > > some effort toward OpenSearch--I say this not understanding exactly how
> > > this sort of thing is considered in a large-scale, world-class software
> > > project like Apache NiFi. (I'm not a contributor, just a grateful
> > > consumer.)
> > >
> > > Russ
> > >
> > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > Along with the itemized list for ancient components we should look at
> > > updating versions of drivers, SDKs, etc. for external systems such as
> > > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0
> > > is probably the right time to get things up to date to make them more
> > > useful to more people.
> > >
> > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <thenatog@gmail.com
> > <mailto:
> > > thenatog@gmail.com>> wrote:
> > > I'm a +1 for removing pretty much all of this stuff. There are security
> > > implications to keeping old dependencies around, so the more old code
> we
> > > can remove the better. I agree that eventually we need to move to
> > > supporting only Java 11+, and as our next release will probably be
> about
> > 4
> > > - 6 months from now that doesn't seem too soon. We could potentially
> > break
> > > this in two and remove the deprecated processors and leave 1.x on Java
> 8,
> > > and finally start on 2.x which would support only Java 11. I'm unsure
> of
> > > what implications changing the date and time handling would have - for
> > > running systems that use long term historical logs, unexpected impacts
> to
> > > time logging could be a problem.
> > >
> > > As Joe says I think feature work will have to be dedicated to 2.x and
> we
> > > could support 1.x for security fixes for some period of time. 2.x seems
> > > like a gargantuan task but it's probably time to get started. Not sure
> > how
> > > we handle all open PRs and the transition between 1.x and 2.x.
> > >
> > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <joe.witt@gmail.com<mailto:
> > > joe.witt@gmail.com>> wrote:
> > >
> > > Jon
> > >
> > > You're right we have to be careful and you're right there are still
> > > significant Java 8 users out there. But we also have to be careful
> > > about security and sustainability of the codebase. If we had talked
> > > about this last year when that article came out I'd have agreed it is
> > > too early. Interestingly that link seems to get updated and I tried
> > > [1] and found more recent data (not sure how recent). Anyway it
> > > suggests Java 8 is still the top dog but we see good growth on 11. In
> > > my $dayjob this aligns to what I'm seeing too. Customers didn't seem
> > > to care about Java 11 until later half last year and now suddenly it
> > > is all over the place.
> > >
> > > I think once we put out a NiFi 2.0 release we'd see rapid decrease in
> > > work on the 1.x line just being blunt. We did this many years ago
> > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
> > > so) but it was purely bug fix/security related bits. We would need to
> > > do something similar. But feature work would almost certainly go to
> > > the 2.x line. Maybe there are other workable models but my instinct
> > > suggests this is likely to follow a similar path.
> > >
> > > ...anyway I agree it isn't that easy of a call to dump Java 8. We
> > > need to make the call in both the interests of the user base and the
> > > contributor base of the community.
> > >
> > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> > >
> > >
> > > Thanks
> > > Joe
> > >
> > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <joe.witt@gmail.com<mailto:
> > > joe.witt@gmail.com>> wrote:
> > > Russ
> > >
> > > Yeah the flow registry is a key part of it. But also now you can
> > > download the flow definition in JSON (upload i think is there now
> > > too). Templates offered a series of challenges such as we store them
> > > in the flow definition which has made flows massive in an unintended
> > > way which isn't fun for cluster behavior.
> > >
> > > We have a couple cases where we headed down a particular concept and
> > > came up with better approaches later. We need to reconcile these with
> > > the benefit of hindsight, and while being careful to be not overly
> > > disruptive to existing users, to reduce the codebase/maintenance
> > > burden and allow continued evolution of the project.
> > >
> > > Thanks
> > >
> > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <russ@windofkeltia.com
> > > <ma...@windofkeltia.com>>
> > > wrote:
> > > Joe,
> > >
> > > I apologize for the off-topic intrusion, but what replaces templates?
> > > The Registry? Templates rocked and we have used them since 0.5.x.
> > >
> > > Russ
> > >
> > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > David,
> > >
> > > I think this is a highly reasonable approach and such a focus will
> > > greatly help make a 2.0 release far more approachable to knock out.
> > > Not only that but tech debt reduction would help make work towards
> > > major features we'd think about in a 'major release' sense more
> > > approachable.
> > >
> > > We should remove all deprecated things (as well as verify we have the
> > > right list). We should remove/consider removal of deprecated
> > > concepts
> > > like templates. We should consider whether we can resolve the
> > > various
> > > ways we've handled what are now parameters down to one clean
> > > approach.
> > > We should remove options in the nifi.properties which turn out to
> > > never be used quite right (if there are). There is quite a bit we
> > > can
> > > do purely in the name of tech debt reduction.
> > >
> > > Lots to consider here but I think this is the right discussion.
> > >
> > > Than ks
> > >
> > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bbende@gmail.com<mailto:
> > > bbende@gmail.com>>
> > > wrote:
> > > I'm a +1 for this... Not sure if this falls under "Removing
> > > Deprecated
> > > Components", but I think we should also look at anything that has
> > > been
> > > marked as deprecated throughout the code base as a candidate for
> > > removal. There are quite a few classes, methods, properties, etc
> > > that
> > > have been waiting for a chance to be removed.
> > >
> > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > <ex...@apache.org>>
> wrote:
> > > Team,
> > >
> > > With all of the excellent work that many have contributed to NiFi
> > > over the
> > > years, the code base has also accumulated some amount of technical
> > > debt. A
> > > handful of components have been marked as deprecated, and some
> > > components
> > > remain in the code base to support integration with old versions
> > > of various
> > > products. Following the principles of semantic versioning,
> > > introducing a
> > > major release would provide the opportunity to remove these
> > > deprecated and
> > > unsupported components.
> > >
> > > Rather than focusing the next major release on new features, what
> > > do you
> > > think about focusing on technical debt removal? This approach
> > > would not
> > > make for the most interesting release, but it provides the
> > > opportunity to
> > > clean up elements that involve breaking changes.
> > >
> > > Focusing on technical debt, at least three primary goals come to
> > > mind for
> > > the next major release:
> > >
> > > 1. Removal of deprecated and unmaintained components
> > > 2. Require Java 11 as the minimum supported version
> > > 3. Transition internal date and time handling to JSR 310 java.time
> > > components
> > >
> > > *Removing Deprecated Components*
> > >
> > > Removing support for older and deprecated components provides a
> > > great
> > > opportunity to improve the overall security posture when it comes
> > > to
> > > maintaining dependencies. The OWASP dependency plugin report
> > > currently
> > > generates 50 MB of HTML for questionable dependencies, many of
> > > which are
> > > related to old versions of various libraries.
> > >
> > > As a starting point, here are a handful of components and
> > > extension modules
> > > that could be targeted for removal in a major version:
> > >
> > > - PostHTTP and GetHTTP
> > > - ListenLumberjack and the entire nifi-lumberjack-bundle
> > > - ListenBeats and the entire nifi-beats-bundle
> > > - Elasticsearch 5 components
> > > - Hive 1 and 2 components
> > >
> > > *Requiring Java 11*
> > >
> > > Java 8 is now over seven years old, and NiFi has supported general
> > > compatibility with Java 11 for several years. NiFi 1.14.0
> > > incorporated
> > > internal improvements specifically related to TLS 1.3, which
> > > allowed
> > > closing out the long-running Java 11 compatibility epic NIFI-5174.
> > > Making
> > > Java 11 the minimum required version provides the opportunity to
> > > address
> > > any lingering edge cases and put NiFi in a better position to
> > > support
> > > current Java versions.
> > >
> > > *JSR 310 for Date and Time Handling*
> > >
> > > Without making the scope too broad, transitioning internal date
> > > and time
> > > handling to use DateTimeFormatter instead of SimpleDateFormat
> > > would provide
> > > a number of advantages. The Java Time components provide much
> > > better
> > > clarity when it comes to handling localized date and time
> > > representations,
> > > and also avoid the inherent confusion of java.sql.Date extending
> > > java.util.Date. Many internal components, specifically
> > > Record-oriented
> > > processors and services, rely on date parsing, leading to
> > > confusion and
> > > various workarounds. The pattern formats of SimpleDateFormat and
> > > DateTimeFormatter are very similar, but there are a few subtle
> > > differences.
> > > Making this transition would provide a much better foundation
> > > going forward.
> > > *Conclusion*
> > >
> > > Thanks for giving this proposal some consideration. Many of you
> > > have been
> > > developing NiFi for years and I look forward to your feedback. I
> > > would be
> > > glad to put together a more formalized recommendation on
> > > Confluence and
> > > write up Jira epics if this general approach sounds agreeable to
> > > the
> > > community.
> > >
> > > Regards,
> > > David Handermann
> > >
> > >
> > >
> >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Chris Sampson <ch...@naimuri.com.INVALID>.
Might be worth considering refactoring the build as part of this work too,
e.g. only building the bits of the repo affected by a commit, etc. -
discussed briefly in previous threads but don't think any changes made yet.
If NARs/components are likely to be split up and refactored then such work
around the build probably makes sense to consider.
I've a couple of PRs open that include updates to Elasticsearch versions
already, although I stopped at 7.10.2 (after which Elastic changed licence)
in case there were licence concerns. But more work can be done to tidy up
the processors, absolutely.
AWS libraries to v2 would seem a sensible move and a refactor of those
processors as well.
Cheers,
Chris Sampson
On Sat, 24 Jul 2021, 17:47 David Handermann, <ex...@apache.org>
wrote:
> Thanks for pointing out the standard NAR bundles, Mark. There are a number
> of components in the standard NAR bundles with particular dependencies that
> would make more sense in separate NARs. Reorganizing the standard NAR to
> components with limited dependencies and wide applicability would
> definitely help with future maintenance.
>
> Regards,
> David Handermann
>
> On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <ma...@hotmail.com> wrote:
>
> > There’s also some code that exists in order to maintain backward
> > compatibility in the repositories. I would very much like the
> repositories
> > to contain no unnecessary code. And swap file format supports really old
> > formats. And the old impls of the repositories themselves, like
> > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff
> there
> > that can be removed. And some methods in ProcessSession that are never
> used
> > by any processor in the codebase but exists in the public API so can’t be
> > removed till 2.0.
> >
> > I think his is also a great time to clean up the “standard nar.” At this
> > point, it’s something like 70 MB. And many of the components there are
> not
> > really “standard” - things like connecting to FTP & SFTP servers, XML
> > processing, Jolt transform, etc. could potentially be moved into other
> > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is 6.9 MB is
> not
> > necessary for stateless or minifi java. Lots of things probably to
> > reconsider within the standard nar.
> >
> > I definitely think this is a reasonable approach, to allow for a 2.0 that
> > is not a huge feature release but allows the project to be simpler and
> more
> > nimble.
> >
> > Thanks
> > -Mark
> >
> > On Jul 24, 2021, at 10:59 AM, Mike Thomsen <mikerthomsen@gmail.com
> <mailto:
> > mikerthomsen@gmail.com>> wrote:
> >
> > Russell,
> >
> > AFAICT from looking at Elastic's repos, the low level REST client is
> > still fine.
> >
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> >
> > Our Elasticsearch support is spread over two NARs at present. One uses
> > OkHttp and the other uses that low level Elastic REST client.
> > Therefore, I think we're fine on licensing for the moment.
> >
> > Mike
> >
> > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <russ@windofkeltia.com
> > <ma...@windofkeltia.com>> wrote:
> >
> > Bringing up Elastic also reminds me that the Elastic framework has just
> > recently transitioned out of Open Source, so to acknowledge that, maybe
> > some effort toward OpenSearch--I say this not understanding exactly how
> > this sort of thing is considered in a large-scale, world-class software
> > project like Apache NiFi. (I'm not a contributor, just a grateful
> > consumer.)
> >
> > Russ
> >
> > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > Along with the itemized list for ancient components we should look at
> > updating versions of drivers, SDKs, etc. for external systems such as
> > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0
> > is probably the right time to get things up to date to make them more
> > useful to more people.
> >
> > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <thenatog@gmail.com
> <mailto:
> > thenatog@gmail.com>> wrote:
> > I'm a +1 for removing pretty much all of this stuff. There are security
> > implications to keeping old dependencies around, so the more old code we
> > can remove the better. I agree that eventually we need to move to
> > supporting only Java 11+, and as our next release will probably be about
> 4
> > - 6 months from now that doesn't seem too soon. We could potentially
> break
> > this in two and remove the deprecated processors and leave 1.x on Java 8,
> > and finally start on 2.x which would support only Java 11. I'm unsure of
> > what implications changing the date and time handling would have - for
> > running systems that use long term historical logs, unexpected impacts to
> > time logging could be a problem.
> >
> > As Joe says I think feature work will have to be dedicated to 2.x and we
> > could support 1.x for security fixes for some period of time. 2.x seems
> > like a gargantuan task but it's probably time to get started. Not sure
> how
> > we handle all open PRs and the transition between 1.x and 2.x.
> >
> > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <joe.witt@gmail.com<mailto:
> > joe.witt@gmail.com>> wrote:
> >
> > Jon
> >
> > You're right we have to be careful and you're right there are still
> > significant Java 8 users out there. But we also have to be careful
> > about security and sustainability of the codebase. If we had talked
> > about this last year when that article came out I'd have agreed it is
> > too early. Interestingly that link seems to get updated and I tried
> > [1] and found more recent data (not sure how recent). Anyway it
> > suggests Java 8 is still the top dog but we see good growth on 11. In
> > my $dayjob this aligns to what I'm seeing too. Customers didn't seem
> > to care about Java 11 until later half last year and now suddenly it
> > is all over the place.
> >
> > I think once we put out a NiFi 2.0 release we'd see rapid decrease in
> > work on the 1.x line just being blunt. We did this many years ago
> > with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
> > so) but it was purely bug fix/security related bits. We would need to
> > do something similar. But feature work would almost certainly go to
> > the 2.x line. Maybe there are other workable models but my instinct
> > suggests this is likely to follow a similar path.
> >
> > ...anyway I agree it isn't that easy of a call to dump Java 8. We
> > need to make the call in both the interests of the user base and the
> > contributor base of the community.
> >
> > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> >
> >
> > Thanks
> > Joe
> >
> > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <joe.witt@gmail.com<mailto:
> > joe.witt@gmail.com>> wrote:
> > Russ
> >
> > Yeah the flow registry is a key part of it. But also now you can
> > download the flow definition in JSON (upload i think is there now
> > too). Templates offered a series of challenges such as we store them
> > in the flow definition which has made flows massive in an unintended
> > way which isn't fun for cluster behavior.
> >
> > We have a couple cases where we headed down a particular concept and
> > came up with better approaches later. We need to reconcile these with
> > the benefit of hindsight, and while being careful to be not overly
> > disruptive to existing users, to reduce the codebase/maintenance
> > burden and allow continued evolution of the project.
> >
> > Thanks
> >
> > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <russ@windofkeltia.com
> > <ma...@windofkeltia.com>>
> > wrote:
> > Joe,
> >
> > I apologize for the off-topic intrusion, but what replaces templates?
> > The Registry? Templates rocked and we have used them since 0.5.x.
> >
> > Russ
> >
> > On 7/23/21 8:31 AM, Joe Witt wrote:
> > David,
> >
> > I think this is a highly reasonable approach and such a focus will
> > greatly help make a 2.0 release far more approachable to knock out.
> > Not only that but tech debt reduction would help make work towards
> > major features we'd think about in a 'major release' sense more
> > approachable.
> >
> > We should remove all deprecated things (as well as verify we have the
> > right list). We should remove/consider removal of deprecated
> > concepts
> > like templates. We should consider whether we can resolve the
> > various
> > ways we've handled what are now parameters down to one clean
> > approach.
> > We should remove options in the nifi.properties which turn out to
> > never be used quite right (if there are). There is quite a bit we
> > can
> > do purely in the name of tech debt reduction.
> >
> > Lots to consider here but I think this is the right discussion.
> >
> > Than ks
> >
> > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bbende@gmail.com<mailto:
> > bbende@gmail.com>>
> > wrote:
> > I'm a +1 for this... Not sure if this falls under "Removing
> > Deprecated
> > Components", but I think we should also look at anything that has
> > been
> > marked as deprecated throughout the code base as a candidate for
> > removal. There are quite a few classes, methods, properties, etc
> > that
> > have been waiting for a chance to be removed.
> >
> > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > <ex...@apache.org>> wrote:
> > Team,
> >
> > With all of the excellent work that many have contributed to NiFi
> > over the
> > years, the code base has also accumulated some amount of technical
> > debt. A
> > handful of components have been marked as deprecated, and some
> > components
> > remain in the code base to support integration with old versions
> > of various
> > products. Following the principles of semantic versioning,
> > introducing a
> > major release would provide the opportunity to remove these
> > deprecated and
> > unsupported components.
> >
> > Rather than focusing the next major release on new features, what
> > do you
> > think about focusing on technical debt removal? This approach
> > would not
> > make for the most interesting release, but it provides the
> > opportunity to
> > clean up elements that involve breaking changes.
> >
> > Focusing on technical debt, at least three primary goals come to
> > mind for
> > the next major release:
> >
> > 1. Removal of deprecated and unmaintained components
> > 2. Require Java 11 as the minimum supported version
> > 3. Transition internal date and time handling to JSR 310 java.time
> > components
> >
> > *Removing Deprecated Components*
> >
> > Removing support for older and deprecated components provides a
> > great
> > opportunity to improve the overall security posture when it comes
> > to
> > maintaining dependencies. The OWASP dependency plugin report
> > currently
> > generates 50 MB of HTML for questionable dependencies, many of
> > which are
> > related to old versions of various libraries.
> >
> > As a starting point, here are a handful of components and
> > extension modules
> > that could be targeted for removal in a major version:
> >
> > - PostHTTP and GetHTTP
> > - ListenLumberjack and the entire nifi-lumberjack-bundle
> > - ListenBeats and the entire nifi-beats-bundle
> > - Elasticsearch 5 components
> > - Hive 1 and 2 components
> >
> > *Requiring Java 11*
> >
> > Java 8 is now over seven years old, and NiFi has supported general
> > compatibility with Java 11 for several years. NiFi 1.14.0
> > incorporated
> > internal improvements specifically related to TLS 1.3, which
> > allowed
> > closing out the long-running Java 11 compatibility epic NIFI-5174.
> > Making
> > Java 11 the minimum required version provides the opportunity to
> > address
> > any lingering edge cases and put NiFi in a better position to
> > support
> > current Java versions.
> >
> > *JSR 310 for Date and Time Handling*
> >
> > Without making the scope too broad, transitioning internal date
> > and time
> > handling to use DateTimeFormatter instead of SimpleDateFormat
> > would provide
> > a number of advantages. The Java Time components provide much
> > better
> > clarity when it comes to handling localized date and time
> > representations,
> > and also avoid the inherent confusion of java.sql.Date extending
> > java.util.Date. Many internal components, specifically
> > Record-oriented
> > processors and services, rely on date parsing, leading to
> > confusion and
> > various workarounds. The pattern formats of SimpleDateFormat and
> > DateTimeFormatter are very similar, but there are a few subtle
> > differences.
> > Making this transition would provide a much better foundation
> > going forward.
> > *Conclusion*
> >
> > Thanks for giving this proposal some consideration. Many of you
> > have been
> > developing NiFi for years and I look forward to your feedback. I
> > would be
> > glad to put together a more formalized recommendation on
> > Confluence and
> > write up Jira epics if this general approach sounds agreeable to
> > the
> > community.
> >
> > Regards,
> > David Handermann
> >
> >
> >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by David Handermann <ex...@apache.org>.
Thanks for pointing out the standard NAR bundles, Mark. There are a number
of components in the standard NAR bundles with particular dependencies that
would make more sense in separate NARs. Reorganizing the standard NAR to
components with limited dependencies and wide applicability would
definitely help with future maintenance.
Regards,
David Handermann
On Sat, Jul 24, 2021 at 10:57 AM Mark Payne <ma...@hotmail.com> wrote:
> There’s also some code that exists in order to maintain backward
> compatibility in the repositories. I would very much like the repositories
> to contain no unnecessary code. And swap file format supports really old
> formats. And the old impls of the repositories themselves, like
> PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff there
> that can be removed. And some methods in ProcessSession that are never used
> by any processor in the codebase but exists in the public API so can’t be
> removed till 2.0.
>
> I think his is also a great time to clean up the “standard nar.” At this
> point, it’s something like 70 MB. And many of the components there are not
> really “standard” - things like connecting to FTP & SFTP servers, XML
> processing, Jolt transform, etc. could potentially be moved into other
> nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is 6.9 MB is not
> necessary for stateless or minifi java. Lots of things probably to
> reconsider within the standard nar.
>
> I definitely think this is a reasonable approach, to allow for a 2.0 that
> is not a huge feature release but allows the project to be simpler and more
> nimble.
>
> Thanks
> -Mark
>
> On Jul 24, 2021, at 10:59 AM, Mike Thomsen <mikerthomsen@gmail.com<mailto:
> mikerthomsen@gmail.com>> wrote:
>
> Russell,
>
> AFAICT from looking at Elastic's repos, the low level REST client is
> still fine.
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
>
> Our Elasticsearch support is spread over two NARs at present. One uses
> OkHttp and the other uses that low level Elastic REST client.
> Therefore, I think we're fine on licensing for the moment.
>
> Mike
>
> On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <russ@windofkeltia.com
> <ma...@windofkeltia.com>> wrote:
>
> Bringing up Elastic also reminds me that the Elastic framework has just
> recently transitioned out of Open Source, so to acknowledge that, maybe
> some effort toward OpenSearch--I say this not understanding exactly how
> this sort of thing is considered in a large-scale, world-class software
> project like Apache NiFi. (I'm not a contributor, just a grateful
> consumer.)
>
> Russ
>
> On 7/23/21 10:28 AM, Matt Burgess wrote:
> Along with the itemized list for ancient components we should look at
> updating versions of drivers, SDKs, etc. for external systems such as
> Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0
> is probably the right time to get things up to date to make them more
> useful to more people.
>
> On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <thenatog@gmail.com<mailto:
> thenatog@gmail.com>> wrote:
> I'm a +1 for removing pretty much all of this stuff. There are security
> implications to keeping old dependencies around, so the more old code we
> can remove the better. I agree that eventually we need to move to
> supporting only Java 11+, and as our next release will probably be about 4
> - 6 months from now that doesn't seem too soon. We could potentially break
> this in two and remove the deprecated processors and leave 1.x on Java 8,
> and finally start on 2.x which would support only Java 11. I'm unsure of
> what implications changing the date and time handling would have - for
> running systems that use long term historical logs, unexpected impacts to
> time logging could be a problem.
>
> As Joe says I think feature work will have to be dedicated to 2.x and we
> could support 1.x for security fixes for some period of time. 2.x seems
> like a gargantuan task but it's probably time to get started. Not sure how
> we handle all open PRs and the transition between 1.x and 2.x.
>
> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <joe.witt@gmail.com<mailto:
> joe.witt@gmail.com>> wrote:
>
> Jon
>
> You're right we have to be careful and you're right there are still
> significant Java 8 users out there. But we also have to be careful
> about security and sustainability of the codebase. If we had talked
> about this last year when that article came out I'd have agreed it is
> too early. Interestingly that link seems to get updated and I tried
> [1] and found more recent data (not sure how recent). Anyway it
> suggests Java 8 is still the top dog but we see good growth on 11. In
> my $dayjob this aligns to what I'm seeing too. Customers didn't seem
> to care about Java 11 until later half last year and now suddenly it
> is all over the place.
>
> I think once we put out a NiFi 2.0 release we'd see rapid decrease in
> work on the 1.x line just being blunt. We did this many years ago
> with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
> so) but it was purely bug fix/security related bits. We would need to
> do something similar. But feature work would almost certainly go to
> the 2.x line. Maybe there are other workable models but my instinct
> suggests this is likely to follow a similar path.
>
> ...anyway I agree it isn't that easy of a call to dump Java 8. We
> need to make the call in both the interests of the user base and the
> contributor base of the community.
>
> [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
>
>
> Thanks
> Joe
>
> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <joe.witt@gmail.com<mailto:
> joe.witt@gmail.com>> wrote:
> Russ
>
> Yeah the flow registry is a key part of it. But also now you can
> download the flow definition in JSON (upload i think is there now
> too). Templates offered a series of challenges such as we store them
> in the flow definition which has made flows massive in an unintended
> way which isn't fun for cluster behavior.
>
> We have a couple cases where we headed down a particular concept and
> came up with better approaches later. We need to reconcile these with
> the benefit of hindsight, and while being careful to be not overly
> disruptive to existing users, to reduce the codebase/maintenance
> burden and allow continued evolution of the project.
>
> Thanks
>
> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <russ@windofkeltia.com
> <ma...@windofkeltia.com>>
> wrote:
> Joe,
>
> I apologize for the off-topic intrusion, but what replaces templates?
> The Registry? Templates rocked and we have used them since 0.5.x.
>
> Russ
>
> On 7/23/21 8:31 AM, Joe Witt wrote:
> David,
>
> I think this is a highly reasonable approach and such a focus will
> greatly help make a 2.0 release far more approachable to knock out.
> Not only that but tech debt reduction would help make work towards
> major features we'd think about in a 'major release' sense more
> approachable.
>
> We should remove all deprecated things (as well as verify we have the
> right list). We should remove/consider removal of deprecated
> concepts
> like templates. We should consider whether we can resolve the
> various
> ways we've handled what are now parameters down to one clean
> approach.
> We should remove options in the nifi.properties which turn out to
> never be used quite right (if there are). There is quite a bit we
> can
> do purely in the name of tech debt reduction.
>
> Lots to consider here but I think this is the right discussion.
>
> Than ks
>
> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bbende@gmail.com<mailto:
> bbende@gmail.com>>
> wrote:
> I'm a +1 for this... Not sure if this falls under "Removing
> Deprecated
> Components", but I think we should also look at anything that has
> been
> marked as deprecated throughout the code base as a candidate for
> removal. There are quite a few classes, methods, properties, etc
> that
> have been waiting for a chance to be removed.
>
> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> <ex...@apache.org>> wrote:
> Team,
>
> With all of the excellent work that many have contributed to NiFi
> over the
> years, the code base has also accumulated some amount of technical
> debt. A
> handful of components have been marked as deprecated, and some
> components
> remain in the code base to support integration with old versions
> of various
> products. Following the principles of semantic versioning,
> introducing a
> major release would provide the opportunity to remove these
> deprecated and
> unsupported components.
>
> Rather than focusing the next major release on new features, what
> do you
> think about focusing on technical debt removal? This approach
> would not
> make for the most interesting release, but it provides the
> opportunity to
> clean up elements that involve breaking changes.
>
> Focusing on technical debt, at least three primary goals come to
> mind for
> the next major release:
>
> 1. Removal of deprecated and unmaintained components
> 2. Require Java 11 as the minimum supported version
> 3. Transition internal date and time handling to JSR 310 java.time
> components
>
> *Removing Deprecated Components*
>
> Removing support for older and deprecated components provides a
> great
> opportunity to improve the overall security posture when it comes
> to
> maintaining dependencies. The OWASP dependency plugin report
> currently
> generates 50 MB of HTML for questionable dependencies, many of
> which are
> related to old versions of various libraries.
>
> As a starting point, here are a handful of components and
> extension modules
> that could be targeted for removal in a major version:
>
> - PostHTTP and GetHTTP
> - ListenLumberjack and the entire nifi-lumberjack-bundle
> - ListenBeats and the entire nifi-beats-bundle
> - Elasticsearch 5 components
> - Hive 1 and 2 components
>
> *Requiring Java 11*
>
> Java 8 is now over seven years old, and NiFi has supported general
> compatibility with Java 11 for several years. NiFi 1.14.0
> incorporated
> internal improvements specifically related to TLS 1.3, which
> allowed
> closing out the long-running Java 11 compatibility epic NIFI-5174.
> Making
> Java 11 the minimum required version provides the opportunity to
> address
> any lingering edge cases and put NiFi in a better position to
> support
> current Java versions.
>
> *JSR 310 for Date and Time Handling*
>
> Without making the scope too broad, transitioning internal date
> and time
> handling to use DateTimeFormatter instead of SimpleDateFormat
> would provide
> a number of advantages. The Java Time components provide much
> better
> clarity when it comes to handling localized date and time
> representations,
> and also avoid the inherent confusion of java.sql.Date extending
> java.util.Date. Many internal components, specifically
> Record-oriented
> processors and services, rely on date parsing, leading to
> confusion and
> various workarounds. The pattern formats of SimpleDateFormat and
> DateTimeFormatter are very similar, but there are a few subtle
> differences.
> Making this transition would provide a much better foundation
> going forward.
> *Conclusion*
>
> Thanks for giving this proposal some consideration. Many of you
> have been
> developing NiFi for years and I look forward to your feedback. I
> would be
> glad to put together a more formalized recommendation on
> Confluence and
> write up Jira epics if this general approach sounds agreeable to
> the
> community.
>
> Regards,
> David Handermann
>
>
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Mark Payne <ma...@hotmail.com>.
There’s also some code that exists in order to maintain backward compatibility in the repositories. I would very much like the repositories to contain no unnecessary code. And swap file format supports really old formats. And the old impls of the repositories themselves, like PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff there that can be removed. And some methods in ProcessSession that are never used by any processor in the codebase but exists in the public API so can’t be removed till 2.0.
I think his is also a great time to clean up the “standard nar.” At this point, it’s something like 70 MB. And many of the components there are not really “standard” - things like connecting to FTP & SFTP servers, XML processing, Jolt transform, etc. could potentially be moved into other nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is 6.9 MB is not necessary for stateless or minifi java. Lots of things probably to reconsider within the standard nar.
I definitely think this is a reasonable approach, to allow for a 2.0 that is not a huge feature release but allows the project to be simpler and more nimble.
Thanks
-Mark
On Jul 24, 2021, at 10:59 AM, Mike Thomsen <mi...@gmail.com>> wrote:
Russell,
AFAICT from looking at Elastic's repos, the low level REST client is
still fine. https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
Our Elasticsearch support is spread over two NARs at present. One uses
OkHttp and the other uses that low level Elastic REST client.
Therefore, I think we're fine on licensing for the moment.
Mike
On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <ru...@windofkeltia.com>> wrote:
Bringing up Elastic also reminds me that the Elastic framework has just
recently transitioned out of Open Source, so to acknowledge that, maybe
some effort toward OpenSearch--I say this not understanding exactly how
this sort of thing is considered in a large-scale, world-class software
project like Apache NiFi. (I'm not a contributor, just a grateful consumer.)
Russ
On 7/23/21 10:28 AM, Matt Burgess wrote:
Along with the itemized list for ancient components we should look at
updating versions of drivers, SDKs, etc. for external systems such as
Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0
is probably the right time to get things up to date to make them more
useful to more people.
On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <th...@gmail.com>> wrote:
I'm a +1 for removing pretty much all of this stuff. There are security
implications to keeping old dependencies around, so the more old code we
can remove the better. I agree that eventually we need to move to
supporting only Java 11+, and as our next release will probably be about 4
- 6 months from now that doesn't seem too soon. We could potentially break
this in two and remove the deprecated processors and leave 1.x on Java 8,
and finally start on 2.x which would support only Java 11. I'm unsure of
what implications changing the date and time handling would have - for
running systems that use long term historical logs, unexpected impacts to
time logging could be a problem.
As Joe says I think feature work will have to be dedicated to 2.x and we
could support 1.x for security fixes for some period of time. 2.x seems
like a gargantuan task but it's probably time to get started. Not sure how
we handle all open PRs and the transition between 1.x and 2.x.
On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <jo...@gmail.com>> wrote:
Jon
You're right we have to be careful and you're right there are still
significant Java 8 users out there. But we also have to be careful
about security and sustainability of the codebase. If we had talked
about this last year when that article came out I'd have agreed it is
too early. Interestingly that link seems to get updated and I tried
[1] and found more recent data (not sure how recent). Anyway it
suggests Java 8 is still the top dog but we see good growth on 11. In
my $dayjob this aligns to what I'm seeing too. Customers didn't seem
to care about Java 11 until later half last year and now suddenly it
is all over the place.
I think once we put out a NiFi 2.0 release we'd see rapid decrease in
work on the 1.x line just being blunt. We did this many years ago
with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
so) but it was purely bug fix/security related bits. We would need to
do something similar. But feature work would almost certainly go to
the 2.x line. Maybe there are other workable models but my instinct
suggests this is likely to follow a similar path.
...anyway I agree it isn't that easy of a call to dump Java 8. We
need to make the call in both the interests of the user base and the
contributor base of the community.
[1] https://www.jetbrains.com/lp/devecosystem-2021/java/
Thanks
Joe
On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <jo...@gmail.com>> wrote:
Russ
Yeah the flow registry is a key part of it. But also now you can
download the flow definition in JSON (upload i think is there now
too). Templates offered a series of challenges such as we store them
in the flow definition which has made flows massive in an unintended
way which isn't fun for cluster behavior.
We have a couple cases where we headed down a particular concept and
came up with better approaches later. We need to reconcile these with
the benefit of hindsight, and while being careful to be not overly
disruptive to existing users, to reduce the codebase/maintenance
burden and allow continued evolution of the project.
Thanks
On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <ru...@windofkeltia.com>>
wrote:
Joe,
I apologize for the off-topic intrusion, but what replaces templates?
The Registry? Templates rocked and we have used them since 0.5.x.
Russ
On 7/23/21 8:31 AM, Joe Witt wrote:
David,
I think this is a highly reasonable approach and such a focus will
greatly help make a 2.0 release far more approachable to knock out.
Not only that but tech debt reduction would help make work towards
major features we'd think about in a 'major release' sense more
approachable.
We should remove all deprecated things (as well as verify we have the
right list). We should remove/consider removal of deprecated
concepts
like templates. We should consider whether we can resolve the
various
ways we've handled what are now parameters down to one clean
approach.
We should remove options in the nifi.properties which turn out to
never be used quite right (if there are). There is quite a bit we
can
do purely in the name of tech debt reduction.
Lots to consider here but I think this is the right discussion.
Than ks
On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bb...@gmail.com>>
wrote:
I'm a +1 for this... Not sure if this falls under "Removing
Deprecated
Components", but I think we should also look at anything that has
been
marked as deprecated throughout the code base as a candidate for
removal. There are quite a few classes, methods, properties, etc
that
have been waiting for a chance to be removed.
On Fri, Jul 23, 2021 at 10:13 AM David Handermann
<ex...@apache.org>> wrote:
Team,
With all of the excellent work that many have contributed to NiFi
over the
years, the code base has also accumulated some amount of technical
debt. A
handful of components have been marked as deprecated, and some
components
remain in the code base to support integration with old versions
of various
products. Following the principles of semantic versioning,
introducing a
major release would provide the opportunity to remove these
deprecated and
unsupported components.
Rather than focusing the next major release on new features, what
do you
think about focusing on technical debt removal? This approach
would not
make for the most interesting release, but it provides the
opportunity to
clean up elements that involve breaking changes.
Focusing on technical debt, at least three primary goals come to
mind for
the next major release:
1. Removal of deprecated and unmaintained components
2. Require Java 11 as the minimum supported version
3. Transition internal date and time handling to JSR 310 java.time
components
*Removing Deprecated Components*
Removing support for older and deprecated components provides a
great
opportunity to improve the overall security posture when it comes
to
maintaining dependencies. The OWASP dependency plugin report
currently
generates 50 MB of HTML for questionable dependencies, many of
which are
related to old versions of various libraries.
As a starting point, here are a handful of components and
extension modules
that could be targeted for removal in a major version:
- PostHTTP and GetHTTP
- ListenLumberjack and the entire nifi-lumberjack-bundle
- ListenBeats and the entire nifi-beats-bundle
- Elasticsearch 5 components
- Hive 1 and 2 components
*Requiring Java 11*
Java 8 is now over seven years old, and NiFi has supported general
compatibility with Java 11 for several years. NiFi 1.14.0
incorporated
internal improvements specifically related to TLS 1.3, which
allowed
closing out the long-running Java 11 compatibility epic NIFI-5174.
Making
Java 11 the minimum required version provides the opportunity to
address
any lingering edge cases and put NiFi in a better position to
support
current Java versions.
*JSR 310 for Date and Time Handling*
Without making the scope too broad, transitioning internal date
and time
handling to use DateTimeFormatter instead of SimpleDateFormat
would provide
a number of advantages. The Java Time components provide much
better
clarity when it comes to handling localized date and time
representations,
and also avoid the inherent confusion of java.sql.Date extending
java.util.Date. Many internal components, specifically
Record-oriented
processors and services, rely on date parsing, leading to
confusion and
various workarounds. The pattern formats of SimpleDateFormat and
DateTimeFormatter are very similar, but there are a few subtle
differences.
Making this transition would provide a much better foundation
going forward.
*Conclusion*
Thanks for giving this proposal some consideration. Many of you
have been
developing NiFi for years and I look forward to your feedback. I
would be
glad to put together a more formalized recommendation on
Confluence and
write up Jira epics if this general approach sounds agreeable to
the
community.
Regards,
David Handermann
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Mike Thomsen <mi...@gmail.com>.
Russell,
AFAICT from looking at Elastic's repos, the low level REST client is
still fine. https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
Our Elasticsearch support is spread over two NARs at present. One uses
OkHttp and the other uses that low level Elastic REST client.
Therefore, I think we're fine on licensing for the moment.
Mike
On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <ru...@windofkeltia.com> wrote:
>
> Bringing up Elastic also reminds me that the Elastic framework has just
> recently transitioned out of Open Source, so to acknowledge that, maybe
> some effort toward OpenSearch--I say this not understanding exactly how
> this sort of thing is considered in a large-scale, world-class software
> project like Apache NiFi. (I'm not a contributor, just a grateful consumer.)
>
> Russ
>
> On 7/23/21 10:28 AM, Matt Burgess wrote:
> > Along with the itemized list for ancient components we should look at
> > updating versions of drivers, SDKs, etc. for external systems such as
> > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0
> > is probably the right time to get things up to date to make them more
> > useful to more people.
> >
> > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <th...@gmail.com> wrote:
> >> I'm a +1 for removing pretty much all of this stuff. There are security
> >> implications to keeping old dependencies around, so the more old code we
> >> can remove the better. I agree that eventually we need to move to
> >> supporting only Java 11+, and as our next release will probably be about 4
> >> - 6 months from now that doesn't seem too soon. We could potentially break
> >> this in two and remove the deprecated processors and leave 1.x on Java 8,
> >> and finally start on 2.x which would support only Java 11. I'm unsure of
> >> what implications changing the date and time handling would have - for
> >> running systems that use long term historical logs, unexpected impacts to
> >> time logging could be a problem.
> >>
> >> As Joe says I think feature work will have to be dedicated to 2.x and we
> >> could support 1.x for security fixes for some period of time. 2.x seems
> >> like a gargantuan task but it's probably time to get started. Not sure how
> >> we handle all open PRs and the transition between 1.x and 2.x.
> >>
> >> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <jo...@gmail.com> wrote:
> >>
> >>> Jon
> >>>
> >>> You're right we have to be careful and you're right there are still
> >>> significant Java 8 users out there. But we also have to be careful
> >>> about security and sustainability of the codebase. If we had talked
> >>> about this last year when that article came out I'd have agreed it is
> >>> too early. Interestingly that link seems to get updated and I tried
> >>> [1] and found more recent data (not sure how recent). Anyway it
> >>> suggests Java 8 is still the top dog but we see good growth on 11. In
> >>> my $dayjob this aligns to what I'm seeing too. Customers didn't seem
> >>> to care about Java 11 until later half last year and now suddenly it
> >>> is all over the place.
> >>>
> >>> I think once we put out a NiFi 2.0 release we'd see rapid decrease in
> >>> work on the 1.x line just being blunt. We did this many years ago
> >>> with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
> >>> so) but it was purely bug fix/security related bits. We would need to
> >>> do something similar. But feature work would almost certainly go to
> >>> the 2.x line. Maybe there are other workable models but my instinct
> >>> suggests this is likely to follow a similar path.
> >>>
> >>> ...anyway I agree it isn't that easy of a call to dump Java 8. We
> >>> need to make the call in both the interests of the user base and the
> >>> contributor base of the community.
> >>>
> >>> [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> >>>
> >>>
> >>> Thanks
> >>> Joe
> >>>
> >>> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <jo...@gmail.com> wrote:
> >>>> Russ
> >>>>
> >>>> Yeah the flow registry is a key part of it. But also now you can
> >>>> download the flow definition in JSON (upload i think is there now
> >>>> too). Templates offered a series of challenges such as we store them
> >>>> in the flow definition which has made flows massive in an unintended
> >>>> way which isn't fun for cluster behavior.
> >>>>
> >>>> We have a couple cases where we headed down a particular concept and
> >>>> came up with better approaches later. We need to reconcile these with
> >>>> the benefit of hindsight, and while being careful to be not overly
> >>>> disruptive to existing users, to reduce the codebase/maintenance
> >>>> burden and allow continued evolution of the project.
> >>>>
> >>>> Thanks
> >>>>
> >>>> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <ru...@windofkeltia.com>
> >>> wrote:
> >>>>> Joe,
> >>>>>
> >>>>> I apologize for the off-topic intrusion, but what replaces templates?
> >>>>> The Registry? Templates rocked and we have used them since 0.5.x.
> >>>>>
> >>>>> Russ
> >>>>>
> >>>>> On 7/23/21 8:31 AM, Joe Witt wrote:
> >>>>>> David,
> >>>>>>
> >>>>>> I think this is a highly reasonable approach and such a focus will
> >>>>>> greatly help make a 2.0 release far more approachable to knock out.
> >>>>>> Not only that but tech debt reduction would help make work towards
> >>>>>> major features we'd think about in a 'major release' sense more
> >>>>>> approachable.
> >>>>>>
> >>>>>> We should remove all deprecated things (as well as verify we have the
> >>>>>> right list). We should remove/consider removal of deprecated
> >>> concepts
> >>>>>> like templates. We should consider whether we can resolve the
> >>> various
> >>>>>> ways we've handled what are now parameters down to one clean
> >>> approach.
> >>>>>> We should remove options in the nifi.properties which turn out to
> >>>>>> never be used quite right (if there are). There is quite a bit we
> >>> can
> >>>>>> do purely in the name of tech debt reduction.
> >>>>>>
> >>>>>> Lots to consider here but I think this is the right discussion.
> >>>>>>
> >>>>>> Than ks
> >>>>>>
> >>>>>> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bb...@gmail.com>
> >>> wrote:
> >>>>>>> I'm a +1 for this... Not sure if this falls under "Removing
> >>> Deprecated
> >>>>>>> Components", but I think we should also look at anything that has
> >>> been
> >>>>>>> marked as deprecated throughout the code base as a candidate for
> >>>>>>> removal. There are quite a few classes, methods, properties, etc
> >>> that
> >>>>>>> have been waiting for a chance to be removed.
> >>>>>>>
> >>>>>>> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> >>>>>>> <ex...@apache.org> wrote:
> >>>>>>>> Team,
> >>>>>>>>
> >>>>>>>> With all of the excellent work that many have contributed to NiFi
> >>> over the
> >>>>>>>> years, the code base has also accumulated some amount of technical
> >>> debt. A
> >>>>>>>> handful of components have been marked as deprecated, and some
> >>> components
> >>>>>>>> remain in the code base to support integration with old versions
> >>> of various
> >>>>>>>> products. Following the principles of semantic versioning,
> >>> introducing a
> >>>>>>>> major release would provide the opportunity to remove these
> >>> deprecated and
> >>>>>>>> unsupported components.
> >>>>>>>>
> >>>>>>>> Rather than focusing the next major release on new features, what
> >>> do you
> >>>>>>>> think about focusing on technical debt removal? This approach
> >>> would not
> >>>>>>>> make for the most interesting release, but it provides the
> >>> opportunity to
> >>>>>>>> clean up elements that involve breaking changes.
> >>>>>>>>
> >>>>>>>> Focusing on technical debt, at least three primary goals come to
> >>> mind for
> >>>>>>>> the next major release:
> >>>>>>>>
> >>>>>>>> 1. Removal of deprecated and unmaintained components
> >>>>>>>> 2. Require Java 11 as the minimum supported version
> >>>>>>>> 3. Transition internal date and time handling to JSR 310 java.time
> >>>>>>>> components
> >>>>>>>>
> >>>>>>>> *Removing Deprecated Components*
> >>>>>>>>
> >>>>>>>> Removing support for older and deprecated components provides a
> >>> great
> >>>>>>>> opportunity to improve the overall security posture when it comes
> >>> to
> >>>>>>>> maintaining dependencies. The OWASP dependency plugin report
> >>> currently
> >>>>>>>> generates 50 MB of HTML for questionable dependencies, many of
> >>> which are
> >>>>>>>> related to old versions of various libraries.
> >>>>>>>>
> >>>>>>>> As a starting point, here are a handful of components and
> >>> extension modules
> >>>>>>>> that could be targeted for removal in a major version:
> >>>>>>>>
> >>>>>>>> - PostHTTP and GetHTTP
> >>>>>>>> - ListenLumberjack and the entire nifi-lumberjack-bundle
> >>>>>>>> - ListenBeats and the entire nifi-beats-bundle
> >>>>>>>> - Elasticsearch 5 components
> >>>>>>>> - Hive 1 and 2 components
> >>>>>>>>
> >>>>>>>> *Requiring Java 11*
> >>>>>>>>
> >>>>>>>> Java 8 is now over seven years old, and NiFi has supported general
> >>>>>>>> compatibility with Java 11 for several years. NiFi 1.14.0
> >>> incorporated
> >>>>>>>> internal improvements specifically related to TLS 1.3, which
> >>> allowed
> >>>>>>>> closing out the long-running Java 11 compatibility epic NIFI-5174.
> >>> Making
> >>>>>>>> Java 11 the minimum required version provides the opportunity to
> >>> address
> >>>>>>>> any lingering edge cases and put NiFi in a better position to
> >>> support
> >>>>>>>> current Java versions.
> >>>>>>>>
> >>>>>>>> *JSR 310 for Date and Time Handling*
> >>>>>>>>
> >>>>>>>> Without making the scope too broad, transitioning internal date
> >>> and time
> >>>>>>>> handling to use DateTimeFormatter instead of SimpleDateFormat
> >>> would provide
> >>>>>>>> a number of advantages. The Java Time components provide much
> >>> better
> >>>>>>>> clarity when it comes to handling localized date and time
> >>> representations,
> >>>>>>>> and also avoid the inherent confusion of java.sql.Date extending
> >>>>>>>> java.util.Date. Many internal components, specifically
> >>> Record-oriented
> >>>>>>>> processors and services, rely on date parsing, leading to
> >>> confusion and
> >>>>>>>> various workarounds. The pattern formats of SimpleDateFormat and
> >>>>>>>> DateTimeFormatter are very similar, but there are a few subtle
> >>> differences.
> >>>>>>>> Making this transition would provide a much better foundation
> >>> going forward.
> >>>>>>>> *Conclusion*
> >>>>>>>>
> >>>>>>>> Thanks for giving this proposal some consideration. Many of you
> >>> have been
> >>>>>>>> developing NiFi for years and I look forward to your feedback. I
> >>> would be
> >>>>>>>> glad to put together a more formalized recommendation on
> >>> Confluence and
> >>>>>>>> write up Jira epics if this general approach sounds agreeable to
> >>> the
> >>>>>>>> community.
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> David Handermann
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Russell Bateman <ru...@windofkeltia.com>.
Bringing up Elastic also reminds me that the Elastic framework has just
recently transitioned out of Open Source, so to acknowledge that, maybe
some effort toward OpenSearch--I say this not understanding exactly how
this sort of thing is considered in a large-scale, world-class software
project like Apache NiFi. (I'm not a contributor, just a grateful consumer.)
Russ
On 7/23/21 10:28 AM, Matt Burgess wrote:
> Along with the itemized list for ancient components we should look at
> updating versions of drivers, SDKs, etc. for external systems such as
> Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0
> is probably the right time to get things up to date to make them more
> useful to more people.
>
> On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <th...@gmail.com> wrote:
>> I'm a +1 for removing pretty much all of this stuff. There are security
>> implications to keeping old dependencies around, so the more old code we
>> can remove the better. I agree that eventually we need to move to
>> supporting only Java 11+, and as our next release will probably be about 4
>> - 6 months from now that doesn't seem too soon. We could potentially break
>> this in two and remove the deprecated processors and leave 1.x on Java 8,
>> and finally start on 2.x which would support only Java 11. I'm unsure of
>> what implications changing the date and time handling would have - for
>> running systems that use long term historical logs, unexpected impacts to
>> time logging could be a problem.
>>
>> As Joe says I think feature work will have to be dedicated to 2.x and we
>> could support 1.x for security fixes for some period of time. 2.x seems
>> like a gargantuan task but it's probably time to get started. Not sure how
>> we handle all open PRs and the transition between 1.x and 2.x.
>>
>> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <jo...@gmail.com> wrote:
>>
>>> Jon
>>>
>>> You're right we have to be careful and you're right there are still
>>> significant Java 8 users out there. But we also have to be careful
>>> about security and sustainability of the codebase. If we had talked
>>> about this last year when that article came out I'd have agreed it is
>>> too early. Interestingly that link seems to get updated and I tried
>>> [1] and found more recent data (not sure how recent). Anyway it
>>> suggests Java 8 is still the top dog but we see good growth on 11. In
>>> my $dayjob this aligns to what I'm seeing too. Customers didn't seem
>>> to care about Java 11 until later half last year and now suddenly it
>>> is all over the place.
>>>
>>> I think once we put out a NiFi 2.0 release we'd see rapid decrease in
>>> work on the 1.x line just being blunt. We did this many years ago
>>> with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
>>> so) but it was purely bug fix/security related bits. We would need to
>>> do something similar. But feature work would almost certainly go to
>>> the 2.x line. Maybe there are other workable models but my instinct
>>> suggests this is likely to follow a similar path.
>>>
>>> ...anyway I agree it isn't that easy of a call to dump Java 8. We
>>> need to make the call in both the interests of the user base and the
>>> contributor base of the community.
>>>
>>> [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
>>>
>>>
>>> Thanks
>>> Joe
>>>
>>> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <jo...@gmail.com> wrote:
>>>> Russ
>>>>
>>>> Yeah the flow registry is a key part of it. But also now you can
>>>> download the flow definition in JSON (upload i think is there now
>>>> too). Templates offered a series of challenges such as we store them
>>>> in the flow definition which has made flows massive in an unintended
>>>> way which isn't fun for cluster behavior.
>>>>
>>>> We have a couple cases where we headed down a particular concept and
>>>> came up with better approaches later. We need to reconcile these with
>>>> the benefit of hindsight, and while being careful to be not overly
>>>> disruptive to existing users, to reduce the codebase/maintenance
>>>> burden and allow continued evolution of the project.
>>>>
>>>> Thanks
>>>>
>>>> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <ru...@windofkeltia.com>
>>> wrote:
>>>>> Joe,
>>>>>
>>>>> I apologize for the off-topic intrusion, but what replaces templates?
>>>>> The Registry? Templates rocked and we have used them since 0.5.x.
>>>>>
>>>>> Russ
>>>>>
>>>>> On 7/23/21 8:31 AM, Joe Witt wrote:
>>>>>> David,
>>>>>>
>>>>>> I think this is a highly reasonable approach and such a focus will
>>>>>> greatly help make a 2.0 release far more approachable to knock out.
>>>>>> Not only that but tech debt reduction would help make work towards
>>>>>> major features we'd think about in a 'major release' sense more
>>>>>> approachable.
>>>>>>
>>>>>> We should remove all deprecated things (as well as verify we have the
>>>>>> right list). We should remove/consider removal of deprecated
>>> concepts
>>>>>> like templates. We should consider whether we can resolve the
>>> various
>>>>>> ways we've handled what are now parameters down to one clean
>>> approach.
>>>>>> We should remove options in the nifi.properties which turn out to
>>>>>> never be used quite right (if there are). There is quite a bit we
>>> can
>>>>>> do purely in the name of tech debt reduction.
>>>>>>
>>>>>> Lots to consider here but I think this is the right discussion.
>>>>>>
>>>>>> Than ks
>>>>>>
>>>>>> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bb...@gmail.com>
>>> wrote:
>>>>>>> I'm a +1 for this... Not sure if this falls under "Removing
>>> Deprecated
>>>>>>> Components", but I think we should also look at anything that has
>>> been
>>>>>>> marked as deprecated throughout the code base as a candidate for
>>>>>>> removal. There are quite a few classes, methods, properties, etc
>>> that
>>>>>>> have been waiting for a chance to be removed.
>>>>>>>
>>>>>>> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
>>>>>>> <ex...@apache.org> wrote:
>>>>>>>> Team,
>>>>>>>>
>>>>>>>> With all of the excellent work that many have contributed to NiFi
>>> over the
>>>>>>>> years, the code base has also accumulated some amount of technical
>>> debt. A
>>>>>>>> handful of components have been marked as deprecated, and some
>>> components
>>>>>>>> remain in the code base to support integration with old versions
>>> of various
>>>>>>>> products. Following the principles of semantic versioning,
>>> introducing a
>>>>>>>> major release would provide the opportunity to remove these
>>> deprecated and
>>>>>>>> unsupported components.
>>>>>>>>
>>>>>>>> Rather than focusing the next major release on new features, what
>>> do you
>>>>>>>> think about focusing on technical debt removal? This approach
>>> would not
>>>>>>>> make for the most interesting release, but it provides the
>>> opportunity to
>>>>>>>> clean up elements that involve breaking changes.
>>>>>>>>
>>>>>>>> Focusing on technical debt, at least three primary goals come to
>>> mind for
>>>>>>>> the next major release:
>>>>>>>>
>>>>>>>> 1. Removal of deprecated and unmaintained components
>>>>>>>> 2. Require Java 11 as the minimum supported version
>>>>>>>> 3. Transition internal date and time handling to JSR 310 java.time
>>>>>>>> components
>>>>>>>>
>>>>>>>> *Removing Deprecated Components*
>>>>>>>>
>>>>>>>> Removing support for older and deprecated components provides a
>>> great
>>>>>>>> opportunity to improve the overall security posture when it comes
>>> to
>>>>>>>> maintaining dependencies. The OWASP dependency plugin report
>>> currently
>>>>>>>> generates 50 MB of HTML for questionable dependencies, many of
>>> which are
>>>>>>>> related to old versions of various libraries.
>>>>>>>>
>>>>>>>> As a starting point, here are a handful of components and
>>> extension modules
>>>>>>>> that could be targeted for removal in a major version:
>>>>>>>>
>>>>>>>> - PostHTTP and GetHTTP
>>>>>>>> - ListenLumberjack and the entire nifi-lumberjack-bundle
>>>>>>>> - ListenBeats and the entire nifi-beats-bundle
>>>>>>>> - Elasticsearch 5 components
>>>>>>>> - Hive 1 and 2 components
>>>>>>>>
>>>>>>>> *Requiring Java 11*
>>>>>>>>
>>>>>>>> Java 8 is now over seven years old, and NiFi has supported general
>>>>>>>> compatibility with Java 11 for several years. NiFi 1.14.0
>>> incorporated
>>>>>>>> internal improvements specifically related to TLS 1.3, which
>>> allowed
>>>>>>>> closing out the long-running Java 11 compatibility epic NIFI-5174.
>>> Making
>>>>>>>> Java 11 the minimum required version provides the opportunity to
>>> address
>>>>>>>> any lingering edge cases and put NiFi in a better position to
>>> support
>>>>>>>> current Java versions.
>>>>>>>>
>>>>>>>> *JSR 310 for Date and Time Handling*
>>>>>>>>
>>>>>>>> Without making the scope too broad, transitioning internal date
>>> and time
>>>>>>>> handling to use DateTimeFormatter instead of SimpleDateFormat
>>> would provide
>>>>>>>> a number of advantages. The Java Time components provide much
>>> better
>>>>>>>> clarity when it comes to handling localized date and time
>>> representations,
>>>>>>>> and also avoid the inherent confusion of java.sql.Date extending
>>>>>>>> java.util.Date. Many internal components, specifically
>>> Record-oriented
>>>>>>>> processors and services, rely on date parsing, leading to
>>> confusion and
>>>>>>>> various workarounds. The pattern formats of SimpleDateFormat and
>>>>>>>> DateTimeFormatter are very similar, but there are a few subtle
>>> differences.
>>>>>>>> Making this transition would provide a much better foundation
>>> going forward.
>>>>>>>> *Conclusion*
>>>>>>>>
>>>>>>>> Thanks for giving this proposal some consideration. Many of you
>>> have been
>>>>>>>> developing NiFi for years and I look forward to your feedback. I
>>> would be
>>>>>>>> glad to put together a more formalized recommendation on
>>> Confluence and
>>>>>>>> write up Jira epics if this general approach sounds agreeable to
>>> the
>>>>>>>> community.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> David Handermann
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Matt Burgess <ma...@apache.org>.
Along with the itemized list for ancient components we should look at
updating versions of drivers, SDKs, etc. for external systems such as
Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0
is probably the right time to get things up to date to make them more
useful to more people.
On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <th...@gmail.com> wrote:
>
> I'm a +1 for removing pretty much all of this stuff. There are security
> implications to keeping old dependencies around, so the more old code we
> can remove the better. I agree that eventually we need to move to
> supporting only Java 11+, and as our next release will probably be about 4
> - 6 months from now that doesn't seem too soon. We could potentially break
> this in two and remove the deprecated processors and leave 1.x on Java 8,
> and finally start on 2.x which would support only Java 11. I'm unsure of
> what implications changing the date and time handling would have - for
> running systems that use long term historical logs, unexpected impacts to
> time logging could be a problem.
>
> As Joe says I think feature work will have to be dedicated to 2.x and we
> could support 1.x for security fixes for some period of time. 2.x seems
> like a gargantuan task but it's probably time to get started. Not sure how
> we handle all open PRs and the transition between 1.x and 2.x.
>
> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <jo...@gmail.com> wrote:
>
> > Jon
> >
> > You're right we have to be careful and you're right there are still
> > significant Java 8 users out there. But we also have to be careful
> > about security and sustainability of the codebase. If we had talked
> > about this last year when that article came out I'd have agreed it is
> > too early. Interestingly that link seems to get updated and I tried
> > [1] and found more recent data (not sure how recent). Anyway it
> > suggests Java 8 is still the top dog but we see good growth on 11. In
> > my $dayjob this aligns to what I'm seeing too. Customers didn't seem
> > to care about Java 11 until later half last year and now suddenly it
> > is all over the place.
> >
> > I think once we put out a NiFi 2.0 release we'd see rapid decrease in
> > work on the 1.x line just being blunt. We did this many years ago
> > with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
> > so) but it was purely bug fix/security related bits. We would need to
> > do something similar. But feature work would almost certainly go to
> > the 2.x line. Maybe there are other workable models but my instinct
> > suggests this is likely to follow a similar path.
> >
> > ...anyway I agree it isn't that easy of a call to dump Java 8. We
> > need to make the call in both the interests of the user base and the
> > contributor base of the community.
> >
> > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> >
> >
> > Thanks
> > Joe
> >
> > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <jo...@gmail.com> wrote:
> > >
> > > Russ
> > >
> > > Yeah the flow registry is a key part of it. But also now you can
> > > download the flow definition in JSON (upload i think is there now
> > > too). Templates offered a series of challenges such as we store them
> > > in the flow definition which has made flows massive in an unintended
> > > way which isn't fun for cluster behavior.
> > >
> > > We have a couple cases where we headed down a particular concept and
> > > came up with better approaches later. We need to reconcile these with
> > > the benefit of hindsight, and while being careful to be not overly
> > > disruptive to existing users, to reduce the codebase/maintenance
> > > burden and allow continued evolution of the project.
> > >
> > > Thanks
> > >
> > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <ru...@windofkeltia.com>
> > wrote:
> > > >
> > > > Joe,
> > > >
> > > > I apologize for the off-topic intrusion, but what replaces templates?
> > > > The Registry? Templates rocked and we have used them since 0.5.x.
> > > >
> > > > Russ
> > > >
> > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > David,
> > > > >
> > > > > I think this is a highly reasonable approach and such a focus will
> > > > > greatly help make a 2.0 release far more approachable to knock out.
> > > > > Not only that but tech debt reduction would help make work towards
> > > > > major features we'd think about in a 'major release' sense more
> > > > > approachable.
> > > > >
> > > > > We should remove all deprecated things (as well as verify we have the
> > > > > right list). We should remove/consider removal of deprecated
> > concepts
> > > > > like templates. We should consider whether we can resolve the
> > various
> > > > > ways we've handled what are now parameters down to one clean
> > approach.
> > > > > We should remove options in the nifi.properties which turn out to
> > > > > never be used quite right (if there are). There is quite a bit we
> > can
> > > > > do purely in the name of tech debt reduction.
> > > > >
> > > > > Lots to consider here but I think this is the right discussion.
> > > > >
> > > > > Than ks
> > > > >
> > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bb...@gmail.com>
> > wrote:
> > > > >> I'm a +1 for this... Not sure if this falls under "Removing
> > Deprecated
> > > > >> Components", but I think we should also look at anything that has
> > been
> > > > >> marked as deprecated throughout the code base as a candidate for
> > > > >> removal. There are quite a few classes, methods, properties, etc
> > that
> > > > >> have been waiting for a chance to be removed.
> > > > >>
> > > > >> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > > >> <ex...@apache.org> wrote:
> > > > >>> Team,
> > > > >>>
> > > > >>> With all of the excellent work that many have contributed to NiFi
> > over the
> > > > >>> years, the code base has also accumulated some amount of technical
> > debt. A
> > > > >>> handful of components have been marked as deprecated, and some
> > components
> > > > >>> remain in the code base to support integration with old versions
> > of various
> > > > >>> products. Following the principles of semantic versioning,
> > introducing a
> > > > >>> major release would provide the opportunity to remove these
> > deprecated and
> > > > >>> unsupported components.
> > > > >>>
> > > > >>> Rather than focusing the next major release on new features, what
> > do you
> > > > >>> think about focusing on technical debt removal? This approach
> > would not
> > > > >>> make for the most interesting release, but it provides the
> > opportunity to
> > > > >>> clean up elements that involve breaking changes.
> > > > >>>
> > > > >>> Focusing on technical debt, at least three primary goals come to
> > mind for
> > > > >>> the next major release:
> > > > >>>
> > > > >>> 1. Removal of deprecated and unmaintained components
> > > > >>> 2. Require Java 11 as the minimum supported version
> > > > >>> 3. Transition internal date and time handling to JSR 310 java.time
> > > > >>> components
> > > > >>>
> > > > >>> *Removing Deprecated Components*
> > > > >>>
> > > > >>> Removing support for older and deprecated components provides a
> > great
> > > > >>> opportunity to improve the overall security posture when it comes
> > to
> > > > >>> maintaining dependencies. The OWASP dependency plugin report
> > currently
> > > > >>> generates 50 MB of HTML for questionable dependencies, many of
> > which are
> > > > >>> related to old versions of various libraries.
> > > > >>>
> > > > >>> As a starting point, here are a handful of components and
> > extension modules
> > > > >>> that could be targeted for removal in a major version:
> > > > >>>
> > > > >>> - PostHTTP and GetHTTP
> > > > >>> - ListenLumberjack and the entire nifi-lumberjack-bundle
> > > > >>> - ListenBeats and the entire nifi-beats-bundle
> > > > >>> - Elasticsearch 5 components
> > > > >>> - Hive 1 and 2 components
> > > > >>>
> > > > >>> *Requiring Java 11*
> > > > >>>
> > > > >>> Java 8 is now over seven years old, and NiFi has supported general
> > > > >>> compatibility with Java 11 for several years. NiFi 1.14.0
> > incorporated
> > > > >>> internal improvements specifically related to TLS 1.3, which
> > allowed
> > > > >>> closing out the long-running Java 11 compatibility epic NIFI-5174.
> > Making
> > > > >>> Java 11 the minimum required version provides the opportunity to
> > address
> > > > >>> any lingering edge cases and put NiFi in a better position to
> > support
> > > > >>> current Java versions.
> > > > >>>
> > > > >>> *JSR 310 for Date and Time Handling*
> > > > >>>
> > > > >>> Without making the scope too broad, transitioning internal date
> > and time
> > > > >>> handling to use DateTimeFormatter instead of SimpleDateFormat
> > would provide
> > > > >>> a number of advantages. The Java Time components provide much
> > better
> > > > >>> clarity when it comes to handling localized date and time
> > representations,
> > > > >>> and also avoid the inherent confusion of java.sql.Date extending
> > > > >>> java.util.Date. Many internal components, specifically
> > Record-oriented
> > > > >>> processors and services, rely on date parsing, leading to
> > confusion and
> > > > >>> various workarounds. The pattern formats of SimpleDateFormat and
> > > > >>> DateTimeFormatter are very similar, but there are a few subtle
> > differences.
> > > > >>> Making this transition would provide a much better foundation
> > going forward.
> > > > >>>
> > > > >>> *Conclusion*
> > > > >>>
> > > > >>> Thanks for giving this proposal some consideration. Many of you
> > have been
> > > > >>> developing NiFi for years and I look forward to your feedback. I
> > would be
> > > > >>> glad to put together a more formalized recommendation on
> > Confluence and
> > > > >>> write up Jira epics if this general approach sounds agreeable to
> > the
> > > > >>> community.
> > > > >>>
> > > > >>> Regards,
> > > > >>> David Handermann
> > > >
> >
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Nathan Gough <th...@gmail.com>.
I'm a +1 for removing pretty much all of this stuff. There are security
implications to keeping old dependencies around, so the more old code we
can remove the better. I agree that eventually we need to move to
supporting only Java 11+, and as our next release will probably be about 4
- 6 months from now that doesn't seem too soon. We could potentially break
this in two and remove the deprecated processors and leave 1.x on Java 8,
and finally start on 2.x which would support only Java 11. I'm unsure of
what implications changing the date and time handling would have - for
running systems that use long term historical logs, unexpected impacts to
time logging could be a problem.
As Joe says I think feature work will have to be dedicated to 2.x and we
could support 1.x for security fixes for some period of time. 2.x seems
like a gargantuan task but it's probably time to get started. Not sure how
we handle all open PRs and the transition between 1.x and 2.x.
On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <jo...@gmail.com> wrote:
> Jon
>
> You're right we have to be careful and you're right there are still
> significant Java 8 users out there. But we also have to be careful
> about security and sustainability of the codebase. If we had talked
> about this last year when that article came out I'd have agreed it is
> too early. Interestingly that link seems to get updated and I tried
> [1] and found more recent data (not sure how recent). Anyway it
> suggests Java 8 is still the top dog but we see good growth on 11. In
> my $dayjob this aligns to what I'm seeing too. Customers didn't seem
> to care about Java 11 until later half last year and now suddenly it
> is all over the place.
>
> I think once we put out a NiFi 2.0 release we'd see rapid decrease in
> work on the 1.x line just being blunt. We did this many years ago
> with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
> so) but it was purely bug fix/security related bits. We would need to
> do something similar. But feature work would almost certainly go to
> the 2.x line. Maybe there are other workable models but my instinct
> suggests this is likely to follow a similar path.
>
> ...anyway I agree it isn't that easy of a call to dump Java 8. We
> need to make the call in both the interests of the user base and the
> contributor base of the community.
>
> [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
>
>
> Thanks
> Joe
>
> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <jo...@gmail.com> wrote:
> >
> > Russ
> >
> > Yeah the flow registry is a key part of it. But also now you can
> > download the flow definition in JSON (upload i think is there now
> > too). Templates offered a series of challenges such as we store them
> > in the flow definition which has made flows massive in an unintended
> > way which isn't fun for cluster behavior.
> >
> > We have a couple cases where we headed down a particular concept and
> > came up with better approaches later. We need to reconcile these with
> > the benefit of hindsight, and while being careful to be not overly
> > disruptive to existing users, to reduce the codebase/maintenance
> > burden and allow continued evolution of the project.
> >
> > Thanks
> >
> > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <ru...@windofkeltia.com>
> wrote:
> > >
> > > Joe,
> > >
> > > I apologize for the off-topic intrusion, but what replaces templates?
> > > The Registry? Templates rocked and we have used them since 0.5.x.
> > >
> > > Russ
> > >
> > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > David,
> > > >
> > > > I think this is a highly reasonable approach and such a focus will
> > > > greatly help make a 2.0 release far more approachable to knock out.
> > > > Not only that but tech debt reduction would help make work towards
> > > > major features we'd think about in a 'major release' sense more
> > > > approachable.
> > > >
> > > > We should remove all deprecated things (as well as verify we have the
> > > > right list). We should remove/consider removal of deprecated
> concepts
> > > > like templates. We should consider whether we can resolve the
> various
> > > > ways we've handled what are now parameters down to one clean
> approach.
> > > > We should remove options in the nifi.properties which turn out to
> > > > never be used quite right (if there are). There is quite a bit we
> can
> > > > do purely in the name of tech debt reduction.
> > > >
> > > > Lots to consider here but I think this is the right discussion.
> > > >
> > > > Than ks
> > > >
> > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bb...@gmail.com>
> wrote:
> > > >> I'm a +1 for this... Not sure if this falls under "Removing
> Deprecated
> > > >> Components", but I think we should also look at anything that has
> been
> > > >> marked as deprecated throughout the code base as a candidate for
> > > >> removal. There are quite a few classes, methods, properties, etc
> that
> > > >> have been waiting for a chance to be removed.
> > > >>
> > > >> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > > >> <ex...@apache.org> wrote:
> > > >>> Team,
> > > >>>
> > > >>> With all of the excellent work that many have contributed to NiFi
> over the
> > > >>> years, the code base has also accumulated some amount of technical
> debt. A
> > > >>> handful of components have been marked as deprecated, and some
> components
> > > >>> remain in the code base to support integration with old versions
> of various
> > > >>> products. Following the principles of semantic versioning,
> introducing a
> > > >>> major release would provide the opportunity to remove these
> deprecated and
> > > >>> unsupported components.
> > > >>>
> > > >>> Rather than focusing the next major release on new features, what
> do you
> > > >>> think about focusing on technical debt removal? This approach
> would not
> > > >>> make for the most interesting release, but it provides the
> opportunity to
> > > >>> clean up elements that involve breaking changes.
> > > >>>
> > > >>> Focusing on technical debt, at least three primary goals come to
> mind for
> > > >>> the next major release:
> > > >>>
> > > >>> 1. Removal of deprecated and unmaintained components
> > > >>> 2. Require Java 11 as the minimum supported version
> > > >>> 3. Transition internal date and time handling to JSR 310 java.time
> > > >>> components
> > > >>>
> > > >>> *Removing Deprecated Components*
> > > >>>
> > > >>> Removing support for older and deprecated components provides a
> great
> > > >>> opportunity to improve the overall security posture when it comes
> to
> > > >>> maintaining dependencies. The OWASP dependency plugin report
> currently
> > > >>> generates 50 MB of HTML for questionable dependencies, many of
> which are
> > > >>> related to old versions of various libraries.
> > > >>>
> > > >>> As a starting point, here are a handful of components and
> extension modules
> > > >>> that could be targeted for removal in a major version:
> > > >>>
> > > >>> - PostHTTP and GetHTTP
> > > >>> - ListenLumberjack and the entire nifi-lumberjack-bundle
> > > >>> - ListenBeats and the entire nifi-beats-bundle
> > > >>> - Elasticsearch 5 components
> > > >>> - Hive 1 and 2 components
> > > >>>
> > > >>> *Requiring Java 11*
> > > >>>
> > > >>> Java 8 is now over seven years old, and NiFi has supported general
> > > >>> compatibility with Java 11 for several years. NiFi 1.14.0
> incorporated
> > > >>> internal improvements specifically related to TLS 1.3, which
> allowed
> > > >>> closing out the long-running Java 11 compatibility epic NIFI-5174.
> Making
> > > >>> Java 11 the minimum required version provides the opportunity to
> address
> > > >>> any lingering edge cases and put NiFi in a better position to
> support
> > > >>> current Java versions.
> > > >>>
> > > >>> *JSR 310 for Date and Time Handling*
> > > >>>
> > > >>> Without making the scope too broad, transitioning internal date
> and time
> > > >>> handling to use DateTimeFormatter instead of SimpleDateFormat
> would provide
> > > >>> a number of advantages. The Java Time components provide much
> better
> > > >>> clarity when it comes to handling localized date and time
> representations,
> > > >>> and also avoid the inherent confusion of java.sql.Date extending
> > > >>> java.util.Date. Many internal components, specifically
> Record-oriented
> > > >>> processors and services, rely on date parsing, leading to
> confusion and
> > > >>> various workarounds. The pattern formats of SimpleDateFormat and
> > > >>> DateTimeFormatter are very similar, but there are a few subtle
> differences.
> > > >>> Making this transition would provide a much better foundation
> going forward.
> > > >>>
> > > >>> *Conclusion*
> > > >>>
> > > >>> Thanks for giving this proposal some consideration. Many of you
> have been
> > > >>> developing NiFi for years and I look forward to your feedback. I
> would be
> > > >>> glad to put together a more formalized recommendation on
> Confluence and
> > > >>> write up Jira epics if this general approach sounds agreeable to
> the
> > > >>> community.
> > > >>>
> > > >>> Regards,
> > > >>> David Handermann
> > >
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Joe Witt <jo...@gmail.com>.
Jon
You're right we have to be careful and you're right there are still
significant Java 8 users out there. But we also have to be careful
about security and sustainability of the codebase. If we had talked
about this last year when that article came out I'd have agreed it is
too early. Interestingly that link seems to get updated and I tried
[1] and found more recent data (not sure how recent). Anyway it
suggests Java 8 is still the top dog but we see good growth on 11. In
my $dayjob this aligns to what I'm seeing too. Customers didn't seem
to care about Java 11 until later half last year and now suddenly it
is all over the place.
I think once we put out a NiFi 2.0 release we'd see rapid decrease in
work on the 1.x line just being blunt. We did this many years ago
with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
so) but it was purely bug fix/security related bits. We would need to
do something similar. But feature work would almost certainly go to
the 2.x line. Maybe there are other workable models but my instinct
suggests this is likely to follow a similar path.
...anyway I agree it isn't that easy of a call to dump Java 8. We
need to make the call in both the interests of the user base and the
contributor base of the community.
[1] https://www.jetbrains.com/lp/devecosystem-2021/java/
Thanks
Joe
On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <jo...@gmail.com> wrote:
>
> Russ
>
> Yeah the flow registry is a key part of it. But also now you can
> download the flow definition in JSON (upload i think is there now
> too). Templates offered a series of challenges such as we store them
> in the flow definition which has made flows massive in an unintended
> way which isn't fun for cluster behavior.
>
> We have a couple cases where we headed down a particular concept and
> came up with better approaches later. We need to reconcile these with
> the benefit of hindsight, and while being careful to be not overly
> disruptive to existing users, to reduce the codebase/maintenance
> burden and allow continued evolution of the project.
>
> Thanks
>
> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <ru...@windofkeltia.com> wrote:
> >
> > Joe,
> >
> > I apologize for the off-topic intrusion, but what replaces templates?
> > The Registry? Templates rocked and we have used them since 0.5.x.
> >
> > Russ
> >
> > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > David,
> > >
> > > I think this is a highly reasonable approach and such a focus will
> > > greatly help make a 2.0 release far more approachable to knock out.
> > > Not only that but tech debt reduction would help make work towards
> > > major features we'd think about in a 'major release' sense more
> > > approachable.
> > >
> > > We should remove all deprecated things (as well as verify we have the
> > > right list). We should remove/consider removal of deprecated concepts
> > > like templates. We should consider whether we can resolve the various
> > > ways we've handled what are now parameters down to one clean approach.
> > > We should remove options in the nifi.properties which turn out to
> > > never be used quite right (if there are). There is quite a bit we can
> > > do purely in the name of tech debt reduction.
> > >
> > > Lots to consider here but I think this is the right discussion.
> > >
> > > Than ks
> > >
> > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bb...@gmail.com> wrote:
> > >> I'm a +1 for this... Not sure if this falls under "Removing Deprecated
> > >> Components", but I think we should also look at anything that has been
> > >> marked as deprecated throughout the code base as a candidate for
> > >> removal. There are quite a few classes, methods, properties, etc that
> > >> have been waiting for a chance to be removed.
> > >>
> > >> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> > >> <ex...@apache.org> wrote:
> > >>> Team,
> > >>>
> > >>> With all of the excellent work that many have contributed to NiFi over the
> > >>> years, the code base has also accumulated some amount of technical debt. A
> > >>> handful of components have been marked as deprecated, and some components
> > >>> remain in the code base to support integration with old versions of various
> > >>> products. Following the principles of semantic versioning, introducing a
> > >>> major release would provide the opportunity to remove these deprecated and
> > >>> unsupported components.
> > >>>
> > >>> Rather than focusing the next major release on new features, what do you
> > >>> think about focusing on technical debt removal? This approach would not
> > >>> make for the most interesting release, but it provides the opportunity to
> > >>> clean up elements that involve breaking changes.
> > >>>
> > >>> Focusing on technical debt, at least three primary goals come to mind for
> > >>> the next major release:
> > >>>
> > >>> 1. Removal of deprecated and unmaintained components
> > >>> 2. Require Java 11 as the minimum supported version
> > >>> 3. Transition internal date and time handling to JSR 310 java.time
> > >>> components
> > >>>
> > >>> *Removing Deprecated Components*
> > >>>
> > >>> Removing support for older and deprecated components provides a great
> > >>> opportunity to improve the overall security posture when it comes to
> > >>> maintaining dependencies. The OWASP dependency plugin report currently
> > >>> generates 50 MB of HTML for questionable dependencies, many of which are
> > >>> related to old versions of various libraries.
> > >>>
> > >>> As a starting point, here are a handful of components and extension modules
> > >>> that could be targeted for removal in a major version:
> > >>>
> > >>> - PostHTTP and GetHTTP
> > >>> - ListenLumberjack and the entire nifi-lumberjack-bundle
> > >>> - ListenBeats and the entire nifi-beats-bundle
> > >>> - Elasticsearch 5 components
> > >>> - Hive 1 and 2 components
> > >>>
> > >>> *Requiring Java 11*
> > >>>
> > >>> Java 8 is now over seven years old, and NiFi has supported general
> > >>> compatibility with Java 11 for several years. NiFi 1.14.0 incorporated
> > >>> internal improvements specifically related to TLS 1.3, which allowed
> > >>> closing out the long-running Java 11 compatibility epic NIFI-5174. Making
> > >>> Java 11 the minimum required version provides the opportunity to address
> > >>> any lingering edge cases and put NiFi in a better position to support
> > >>> current Java versions.
> > >>>
> > >>> *JSR 310 for Date and Time Handling*
> > >>>
> > >>> Without making the scope too broad, transitioning internal date and time
> > >>> handling to use DateTimeFormatter instead of SimpleDateFormat would provide
> > >>> a number of advantages. The Java Time components provide much better
> > >>> clarity when it comes to handling localized date and time representations,
> > >>> and also avoid the inherent confusion of java.sql.Date extending
> > >>> java.util.Date. Many internal components, specifically Record-oriented
> > >>> processors and services, rely on date parsing, leading to confusion and
> > >>> various workarounds. The pattern formats of SimpleDateFormat and
> > >>> DateTimeFormatter are very similar, but there are a few subtle differences.
> > >>> Making this transition would provide a much better foundation going forward.
> > >>>
> > >>> *Conclusion*
> > >>>
> > >>> Thanks for giving this proposal some consideration. Many of you have been
> > >>> developing NiFi for years and I look forward to your feedback. I would be
> > >>> glad to put together a more formalized recommendation on Confluence and
> > >>> write up Jira epics if this general approach sounds agreeable to the
> > >>> community.
> > >>>
> > >>> Regards,
> > >>> David Handermann
> >
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Joe Witt <jo...@gmail.com>.
Russ
Yeah the flow registry is a key part of it. But also now you can
download the flow definition in JSON (upload i think is there now
too). Templates offered a series of challenges such as we store them
in the flow definition which has made flows massive in an unintended
way which isn't fun for cluster behavior.
We have a couple cases where we headed down a particular concept and
came up with better approaches later. We need to reconcile these with
the benefit of hindsight, and while being careful to be not overly
disruptive to existing users, to reduce the codebase/maintenance
burden and allow continued evolution of the project.
Thanks
On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <ru...@windofkeltia.com> wrote:
>
> Joe,
>
> I apologize for the off-topic intrusion, but what replaces templates?
> The Registry? Templates rocked and we have used them since 0.5.x.
>
> Russ
>
> On 7/23/21 8:31 AM, Joe Witt wrote:
> > David,
> >
> > I think this is a highly reasonable approach and such a focus will
> > greatly help make a 2.0 release far more approachable to knock out.
> > Not only that but tech debt reduction would help make work towards
> > major features we'd think about in a 'major release' sense more
> > approachable.
> >
> > We should remove all deprecated things (as well as verify we have the
> > right list). We should remove/consider removal of deprecated concepts
> > like templates. We should consider whether we can resolve the various
> > ways we've handled what are now parameters down to one clean approach.
> > We should remove options in the nifi.properties which turn out to
> > never be used quite right (if there are). There is quite a bit we can
> > do purely in the name of tech debt reduction.
> >
> > Lots to consider here but I think this is the right discussion.
> >
> > Than ks
> >
> > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bb...@gmail.com> wrote:
> >> I'm a +1 for this... Not sure if this falls under "Removing Deprecated
> >> Components", but I think we should also look at anything that has been
> >> marked as deprecated throughout the code base as a candidate for
> >> removal. There are quite a few classes, methods, properties, etc that
> >> have been waiting for a chance to be removed.
> >>
> >> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> >> <ex...@apache.org> wrote:
> >>> Team,
> >>>
> >>> With all of the excellent work that many have contributed to NiFi over the
> >>> years, the code base has also accumulated some amount of technical debt. A
> >>> handful of components have been marked as deprecated, and some components
> >>> remain in the code base to support integration with old versions of various
> >>> products. Following the principles of semantic versioning, introducing a
> >>> major release would provide the opportunity to remove these deprecated and
> >>> unsupported components.
> >>>
> >>> Rather than focusing the next major release on new features, what do you
> >>> think about focusing on technical debt removal? This approach would not
> >>> make for the most interesting release, but it provides the opportunity to
> >>> clean up elements that involve breaking changes.
> >>>
> >>> Focusing on technical debt, at least three primary goals come to mind for
> >>> the next major release:
> >>>
> >>> 1. Removal of deprecated and unmaintained components
> >>> 2. Require Java 11 as the minimum supported version
> >>> 3. Transition internal date and time handling to JSR 310 java.time
> >>> components
> >>>
> >>> *Removing Deprecated Components*
> >>>
> >>> Removing support for older and deprecated components provides a great
> >>> opportunity to improve the overall security posture when it comes to
> >>> maintaining dependencies. The OWASP dependency plugin report currently
> >>> generates 50 MB of HTML for questionable dependencies, many of which are
> >>> related to old versions of various libraries.
> >>>
> >>> As a starting point, here are a handful of components and extension modules
> >>> that could be targeted for removal in a major version:
> >>>
> >>> - PostHTTP and GetHTTP
> >>> - ListenLumberjack and the entire nifi-lumberjack-bundle
> >>> - ListenBeats and the entire nifi-beats-bundle
> >>> - Elasticsearch 5 components
> >>> - Hive 1 and 2 components
> >>>
> >>> *Requiring Java 11*
> >>>
> >>> Java 8 is now over seven years old, and NiFi has supported general
> >>> compatibility with Java 11 for several years. NiFi 1.14.0 incorporated
> >>> internal improvements specifically related to TLS 1.3, which allowed
> >>> closing out the long-running Java 11 compatibility epic NIFI-5174. Making
> >>> Java 11 the minimum required version provides the opportunity to address
> >>> any lingering edge cases and put NiFi in a better position to support
> >>> current Java versions.
> >>>
> >>> *JSR 310 for Date and Time Handling*
> >>>
> >>> Without making the scope too broad, transitioning internal date and time
> >>> handling to use DateTimeFormatter instead of SimpleDateFormat would provide
> >>> a number of advantages. The Java Time components provide much better
> >>> clarity when it comes to handling localized date and time representations,
> >>> and also avoid the inherent confusion of java.sql.Date extending
> >>> java.util.Date. Many internal components, specifically Record-oriented
> >>> processors and services, rely on date parsing, leading to confusion and
> >>> various workarounds. The pattern formats of SimpleDateFormat and
> >>> DateTimeFormatter are very similar, but there are a few subtle differences.
> >>> Making this transition would provide a much better foundation going forward.
> >>>
> >>> *Conclusion*
> >>>
> >>> Thanks for giving this proposal some consideration. Many of you have been
> >>> developing NiFi for years and I look forward to your feedback. I would be
> >>> glad to put together a more formalized recommendation on Confluence and
> >>> write up Jira epics if this general approach sounds agreeable to the
> >>> community.
> >>>
> >>> Regards,
> >>> David Handermann
>
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Russell Bateman <ru...@windofkeltia.com>.
Joe,
I apologize for the off-topic intrusion, but what replaces templates?
The Registry? Templates rocked and we have used them since 0.5.x.
Russ
On 7/23/21 8:31 AM, Joe Witt wrote:
> David,
>
> I think this is a highly reasonable approach and such a focus will
> greatly help make a 2.0 release far more approachable to knock out.
> Not only that but tech debt reduction would help make work towards
> major features we'd think about in a 'major release' sense more
> approachable.
>
> We should remove all deprecated things (as well as verify we have the
> right list). We should remove/consider removal of deprecated concepts
> like templates. We should consider whether we can resolve the various
> ways we've handled what are now parameters down to one clean approach.
> We should remove options in the nifi.properties which turn out to
> never be used quite right (if there are). There is quite a bit we can
> do purely in the name of tech debt reduction.
>
> Lots to consider here but I think this is the right discussion.
>
> Than ks
>
> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bb...@gmail.com> wrote:
>> I'm a +1 for this... Not sure if this falls under "Removing Deprecated
>> Components", but I think we should also look at anything that has been
>> marked as deprecated throughout the code base as a candidate for
>> removal. There are quite a few classes, methods, properties, etc that
>> have been waiting for a chance to be removed.
>>
>> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
>> <ex...@apache.org> wrote:
>>> Team,
>>>
>>> With all of the excellent work that many have contributed to NiFi over the
>>> years, the code base has also accumulated some amount of technical debt. A
>>> handful of components have been marked as deprecated, and some components
>>> remain in the code base to support integration with old versions of various
>>> products. Following the principles of semantic versioning, introducing a
>>> major release would provide the opportunity to remove these deprecated and
>>> unsupported components.
>>>
>>> Rather than focusing the next major release on new features, what do you
>>> think about focusing on technical debt removal? This approach would not
>>> make for the most interesting release, but it provides the opportunity to
>>> clean up elements that involve breaking changes.
>>>
>>> Focusing on technical debt, at least three primary goals come to mind for
>>> the next major release:
>>>
>>> 1. Removal of deprecated and unmaintained components
>>> 2. Require Java 11 as the minimum supported version
>>> 3. Transition internal date and time handling to JSR 310 java.time
>>> components
>>>
>>> *Removing Deprecated Components*
>>>
>>> Removing support for older and deprecated components provides a great
>>> opportunity to improve the overall security posture when it comes to
>>> maintaining dependencies. The OWASP dependency plugin report currently
>>> generates 50 MB of HTML for questionable dependencies, many of which are
>>> related to old versions of various libraries.
>>>
>>> As a starting point, here are a handful of components and extension modules
>>> that could be targeted for removal in a major version:
>>>
>>> - PostHTTP and GetHTTP
>>> - ListenLumberjack and the entire nifi-lumberjack-bundle
>>> - ListenBeats and the entire nifi-beats-bundle
>>> - Elasticsearch 5 components
>>> - Hive 1 and 2 components
>>>
>>> *Requiring Java 11*
>>>
>>> Java 8 is now over seven years old, and NiFi has supported general
>>> compatibility with Java 11 for several years. NiFi 1.14.0 incorporated
>>> internal improvements specifically related to TLS 1.3, which allowed
>>> closing out the long-running Java 11 compatibility epic NIFI-5174. Making
>>> Java 11 the minimum required version provides the opportunity to address
>>> any lingering edge cases and put NiFi in a better position to support
>>> current Java versions.
>>>
>>> *JSR 310 for Date and Time Handling*
>>>
>>> Without making the scope too broad, transitioning internal date and time
>>> handling to use DateTimeFormatter instead of SimpleDateFormat would provide
>>> a number of advantages. The Java Time components provide much better
>>> clarity when it comes to handling localized date and time representations,
>>> and also avoid the inherent confusion of java.sql.Date extending
>>> java.util.Date. Many internal components, specifically Record-oriented
>>> processors and services, rely on date parsing, leading to confusion and
>>> various workarounds. The pattern formats of SimpleDateFormat and
>>> DateTimeFormatter are very similar, but there are a few subtle differences.
>>> Making this transition would provide a much better foundation going forward.
>>>
>>> *Conclusion*
>>>
>>> Thanks for giving this proposal some consideration. Many of you have been
>>> developing NiFi for years and I look forward to your feedback. I would be
>>> glad to put together a more formalized recommendation on Confluence and
>>> write up Jira epics if this general approach sounds agreeable to the
>>> community.
>>>
>>> Regards,
>>> David Handermann
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Joe Witt <jo...@gmail.com>.
David,
I think this is a highly reasonable approach and such a focus will
greatly help make a 2.0 release far more approachable to knock out.
Not only that but tech debt reduction would help make work towards
major features we'd think about in a 'major release' sense more
approachable.
We should remove all deprecated things (as well as verify we have the
right list). We should remove/consider removal of deprecated concepts
like templates. We should consider whether we can resolve the various
ways we've handled what are now parameters down to one clean approach.
We should remove options in the nifi.properties which turn out to
never be used quite right (if there are). There is quite a bit we can
do purely in the name of tech debt reduction.
Lots to consider here but I think this is the right discussion.
Than ks
On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bb...@gmail.com> wrote:
>
> I'm a +1 for this... Not sure if this falls under "Removing Deprecated
> Components", but I think we should also look at anything that has been
> marked as deprecated throughout the code base as a candidate for
> removal. There are quite a few classes, methods, properties, etc that
> have been waiting for a chance to be removed.
>
> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> <ex...@apache.org> wrote:
> >
> > Team,
> >
> > With all of the excellent work that many have contributed to NiFi over the
> > years, the code base has also accumulated some amount of technical debt. A
> > handful of components have been marked as deprecated, and some components
> > remain in the code base to support integration with old versions of various
> > products. Following the principles of semantic versioning, introducing a
> > major release would provide the opportunity to remove these deprecated and
> > unsupported components.
> >
> > Rather than focusing the next major release on new features, what do you
> > think about focusing on technical debt removal? This approach would not
> > make for the most interesting release, but it provides the opportunity to
> > clean up elements that involve breaking changes.
> >
> > Focusing on technical debt, at least three primary goals come to mind for
> > the next major release:
> >
> > 1. Removal of deprecated and unmaintained components
> > 2. Require Java 11 as the minimum supported version
> > 3. Transition internal date and time handling to JSR 310 java.time
> > components
> >
> > *Removing Deprecated Components*
> >
> > Removing support for older and deprecated components provides a great
> > opportunity to improve the overall security posture when it comes to
> > maintaining dependencies. The OWASP dependency plugin report currently
> > generates 50 MB of HTML for questionable dependencies, many of which are
> > related to old versions of various libraries.
> >
> > As a starting point, here are a handful of components and extension modules
> > that could be targeted for removal in a major version:
> >
> > - PostHTTP and GetHTTP
> > - ListenLumberjack and the entire nifi-lumberjack-bundle
> > - ListenBeats and the entire nifi-beats-bundle
> > - Elasticsearch 5 components
> > - Hive 1 and 2 components
> >
> > *Requiring Java 11*
> >
> > Java 8 is now over seven years old, and NiFi has supported general
> > compatibility with Java 11 for several years. NiFi 1.14.0 incorporated
> > internal improvements specifically related to TLS 1.3, which allowed
> > closing out the long-running Java 11 compatibility epic NIFI-5174. Making
> > Java 11 the minimum required version provides the opportunity to address
> > any lingering edge cases and put NiFi in a better position to support
> > current Java versions.
> >
> > *JSR 310 for Date and Time Handling*
> >
> > Without making the scope too broad, transitioning internal date and time
> > handling to use DateTimeFormatter instead of SimpleDateFormat would provide
> > a number of advantages. The Java Time components provide much better
> > clarity when it comes to handling localized date and time representations,
> > and also avoid the inherent confusion of java.sql.Date extending
> > java.util.Date. Many internal components, specifically Record-oriented
> > processors and services, rely on date parsing, leading to confusion and
> > various workarounds. The pattern formats of SimpleDateFormat and
> > DateTimeFormatter are very similar, but there are a few subtle differences.
> > Making this transition would provide a much better foundation going forward.
> >
> > *Conclusion*
> >
> > Thanks for giving this proposal some consideration. Many of you have been
> > developing NiFi for years and I look forward to your feedback. I would be
> > glad to put together a more formalized recommendation on Confluence and
> > write up Jira epics if this general approach sounds agreeable to the
> > community.
> >
> > Regards,
> > David Handermann
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Bryan Bende <bb...@gmail.com>.
I'm a +1 for this... Not sure if this falls under "Removing Deprecated
Components", but I think we should also look at anything that has been
marked as deprecated throughout the code base as a candidate for
removal. There are quite a few classes, methods, properties, etc that
have been waiting for a chance to be removed.
On Fri, Jul 23, 2021 at 10:13 AM David Handermann
<ex...@apache.org> wrote:
>
> Team,
>
> With all of the excellent work that many have contributed to NiFi over the
> years, the code base has also accumulated some amount of technical debt. A
> handful of components have been marked as deprecated, and some components
> remain in the code base to support integration with old versions of various
> products. Following the principles of semantic versioning, introducing a
> major release would provide the opportunity to remove these deprecated and
> unsupported components.
>
> Rather than focusing the next major release on new features, what do you
> think about focusing on technical debt removal? This approach would not
> make for the most interesting release, but it provides the opportunity to
> clean up elements that involve breaking changes.
>
> Focusing on technical debt, at least three primary goals come to mind for
> the next major release:
>
> 1. Removal of deprecated and unmaintained components
> 2. Require Java 11 as the minimum supported version
> 3. Transition internal date and time handling to JSR 310 java.time
> components
>
> *Removing Deprecated Components*
>
> Removing support for older and deprecated components provides a great
> opportunity to improve the overall security posture when it comes to
> maintaining dependencies. The OWASP dependency plugin report currently
> generates 50 MB of HTML for questionable dependencies, many of which are
> related to old versions of various libraries.
>
> As a starting point, here are a handful of components and extension modules
> that could be targeted for removal in a major version:
>
> - PostHTTP and GetHTTP
> - ListenLumberjack and the entire nifi-lumberjack-bundle
> - ListenBeats and the entire nifi-beats-bundle
> - Elasticsearch 5 components
> - Hive 1 and 2 components
>
> *Requiring Java 11*
>
> Java 8 is now over seven years old, and NiFi has supported general
> compatibility with Java 11 for several years. NiFi 1.14.0 incorporated
> internal improvements specifically related to TLS 1.3, which allowed
> closing out the long-running Java 11 compatibility epic NIFI-5174. Making
> Java 11 the minimum required version provides the opportunity to address
> any lingering edge cases and put NiFi in a better position to support
> current Java versions.
>
> *JSR 310 for Date and Time Handling*
>
> Without making the scope too broad, transitioning internal date and time
> handling to use DateTimeFormatter instead of SimpleDateFormat would provide
> a number of advantages. The Java Time components provide much better
> clarity when it comes to handling localized date and time representations,
> and also avoid the inherent confusion of java.sql.Date extending
> java.util.Date. Many internal components, specifically Record-oriented
> processors and services, rely on date parsing, leading to confusion and
> various workarounds. The pattern formats of SimpleDateFormat and
> DateTimeFormatter are very similar, but there are a few subtle differences.
> Making this transition would provide a much better foundation going forward.
>
> *Conclusion*
>
> Thanks for giving this proposal some consideration. Many of you have been
> developing NiFi for years and I look forward to your feedback. I would be
> glad to put together a more formalized recommendation on Confluence and
> write up Jira epics if this general approach sounds agreeable to the
> community.
>
> Regards,
> David Handermann
Re: [DISCUSS] NiFi 2.0 Release Goals
Posted by Joe Gresock <jg...@gmail.com>.
This sounds like a great path for the 2.0 release. I support goals listed
in the NiFi 2.0 Proposed Release Goals page and look forward to the cleanup!
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon>
Virus-free.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
On Fri, Jul 23, 2021 at 10:13 AM David Handermann <
exceptionfactory@apache.org> wrote:
> Team,
>
> With all of the excellent work that many have contributed to NiFi over the
> years, the code base has also accumulated some amount of technical debt. A
> handful of components have been marked as deprecated, and some components
> remain in the code base to support integration with old versions of various
> products. Following the principles of semantic versioning, introducing a
> major release would provide the opportunity to remove these deprecated and
> unsupported components.
>
> Rather than focusing the next major release on new features, what do you
> think about focusing on technical debt removal? This approach would not
> make for the most interesting release, but it provides the opportunity to
> clean up elements that involve breaking changes.
>
> Focusing on technical debt, at least three primary goals come to mind for
> the next major release:
>
> 1. Removal of deprecated and unmaintained components
> 2. Require Java 11 as the minimum supported version
> 3. Transition internal date and time handling to JSR 310 java.time
> components
>
> *Removing Deprecated Components*
>
> Removing support for older and deprecated components provides a great
> opportunity to improve the overall security posture when it comes to
> maintaining dependencies. The OWASP dependency plugin report currently
> generates 50 MB of HTML for questionable dependencies, many of which are
> related to old versions of various libraries.
>
> As a starting point, here are a handful of components and extension modules
> that could be targeted for removal in a major version:
>
> - PostHTTP and GetHTTP
> - ListenLumberjack and the entire nifi-lumberjack-bundle
> - ListenBeats and the entire nifi-beats-bundle
> - Elasticsearch 5 components
> - Hive 1 and 2 components
>
> *Requiring Java 11*
>
> Java 8 is now over seven years old, and NiFi has supported general
> compatibility with Java 11 for several years. NiFi 1.14.0 incorporated
> internal improvements specifically related to TLS 1.3, which allowed
> closing out the long-running Java 11 compatibility epic NIFI-5174. Making
> Java 11 the minimum required version provides the opportunity to address
> any lingering edge cases and put NiFi in a better position to support
> current Java versions.
>
> *JSR 310 for Date and Time Handling*
>
> Without making the scope too broad, transitioning internal date and time
> handling to use DateTimeFormatter instead of SimpleDateFormat would provide
> a number of advantages. The Java Time components provide much better
> clarity when it comes to handling localized date and time representations,
> and also avoid the inherent confusion of java.sql.Date extending
> java.util.Date. Many internal components, specifically Record-oriented
> processors and services, rely on date parsing, leading to confusion and
> various workarounds. The pattern formats of SimpleDateFormat and
> DateTimeFormatter are very similar, but there are a few subtle differences.
> Making this transition would provide a much better foundation going
> forward.
>
> *Conclusion*
>
> Thanks for giving this proposal some consideration. Many of you have been
> developing NiFi for years and I look forward to your feedback. I would be
> glad to put together a more formalized recommendation on Confluence and
> write up Jira epics if this general approach sounds agreeable to the
> community.
>
> Regards,
> David Handermann
>