Re: [DISCUSS] Provided dependency on bcprov-jdk15on

[Andre <an...@fucs.org>]

+1 as well.

On Tue, Nov 1, 2016 at 2:41 AM, Joe Witt <jo...@gmail.com> wrote:

> Team,
>
> In the 1.x line bcprov made its way into the root of the classpath/lib
> folder to support the encrypted sensitive properties features.  This
> can be reworked to isolate it a bit more if we need to.
>
> However, in [1] it has been observed that we have bcprov dependencies
> in about 53 places with each instance taking about 4MB of space for a
> total hit of about 200MB of bcprov.
>
> [1] proposes to make bcprov-jdk15on and bcpkix-jdk15on part of the
> core provided list of things right along side the
> logback/slf4j/logging interfaces. This means all things will have
> access to these going forward and it means it impacts our version
> compatibility.  Given that it would be a standard/provided thing if we
> want to upgrade versions or swap it out with something else we could
> break extensions that then chose to depend on it.  We are not in
> control of the bcprov api just like we're not in control of the
> logging APIs.  If there was some important security related fix we
> needed from bcprov but changing also pulled in api changes for them it
> could break our extensions.
>
> Even with all this said, given the nature, importance, and size
> benefit, I am in favor of NIFI-2954.  But, would like to highlight
> this in case others have perspective they'd like to share.
>
> [1] https://issues.apache.org/jira/browse/NIFI-2954
>
> Thanks
> Joe
>

Re: [DISCUSS] Provided dependency on bcprov-jdk15on

[Joe Witt <jo...@gmail.com>]

Mike

Yep - that will need to get addressed/improved as well.  First step is
get the root classpath cleaned up again (NIFI-2971) and the next step
is to understand and address the excessive declaration of dependencies
which leads to duplication on such a scale.

Thanks
Joe

On Tue, Nov 1, 2016 at 12:15 PM, Michael Moser <mo...@gmail.com> wrote:
> I would like to point out that some very common dependencies have been
> placed into the nifi-standard-services-api-nar.  This nar is a
> Nar-Dependency-Id of several other nars.  Many of these other nars also
> include the same dependencies, which would not be used by the nar class
> loader.
>
> In particular, nifi-standard-services-api-nar contains nifi-utils,
> nifi-processor-utils, and nifi-security-utils.  It also contains the
> bcprov-jdk15on and bcpkix-jdk15on jars.
>
> Then looking at nifi-standard-nar, nifi-aws-nar, nifi-kafka-0-10-nar and
> more, they also contain these same jars in their bundled-dependencies.
> Their Nar-Dependency-Id is nifi-standard-services-api-nar.
>
> -- Mike
>
>
> On Tue, Nov 1, 2016 at 1:50 PM, Aldrin Piri <al...@gmail.com> wrote:
>
>> I think in light of the fact that the bcprov item was an accidental
>> inclusion and not explicitly meant to be handled as such in the framework
>> reorganizing seems like the best option as well.  The original intent of
>> the PR was to minimize that footprint where we were double dipping given
>> the inclusion of nifi-processor-utils.  However, if we manage to shuffle
>> items around a bit, the net size reduction could ultimately be the same.
>> Most importantly, from a maintenance perspective, bundles will only include
>> the library in the NARs that need it which allows that dependency graph to
>> be significantly more comprehensible than the scope tweaking originally
>> presented.
>>
>> Will this also work to adjust the dependencies of nifi-processor-utils (and
>> potentially nifi-security-utils)?  Currently, the archetype also provides
>> nifi-processor-utils, which could lead to similar conflicts for folks that
>> were looking to make use of different versions of JARs within a given NAR.
>>
>> I think the listing Joe provided in the ticket is great and we should be
>> mindful of any deltas while performing each release.  For the reasons
>> highlighted in this chain and on that issue, one jar inclusion could be
>> quite detrimental and cause some difficult issues to those that have
>> created extensions.
>>
>> On Mon, Oct 31, 2016 at 9:39 PM, Joe Witt <jo...@gmail.com> wrote:
>>
>> > So...I might be a bit of a flip flopper...
>> >
>> > Had some off-list discussions with a couple people and they really
>> > disliked this idea.  They, rightfully, pointed out how this
>> > establishes a problematic precedent, how it is really only being done
>> > to save space, and that while perhaps not able to pair down to 200MB
>> > we can likely do more analysis of the dependency graph to improve the
>> > situation.  The real issue that kicked this off was the finding that
>> > bcprov (and commons-lang3) had made their way into the root of our
>> > classloader (which impacts all classloaders) and this was introduced
>> > with the important NIFI-1831 work.  So I've created [1] to address
>> > that and return the application classloader to the minimal set of jars
>> > necessary which should basically be the runtime application launcher,
>> > the logging libraries, and the nifi-api.
>> >
>> > The ultimate solution to addressing the distribution size concerns
>> > will be to address it via the extension registry.
>> >
>> > [1] https://issues.apache.org/jira/browse/NIFI-2971
>> >
>> > Thanks
>> > Joe
>> >
>> > On Mon, Oct 31, 2016 at 8:51 PM, Andre <an...@fucs.org> wrote:
>> > > +1 as well.
>> > >
>> > > On Tue, Nov 1, 2016 at 2:41 AM, Joe Witt <jo...@gmail.com> wrote:
>> > >
>> > >> Team,
>> > >>
>> > >> In the 1.x line bcprov made its way into the root of the classpath/lib
>> > >> folder to support the encrypted sensitive properties features.  This
>> > >> can be reworked to isolate it a bit more if we need to.
>> > >>
>> > >> However, in [1] it has been observed that we have bcprov dependencies
>> > >> in about 53 places with each instance taking about 4MB of space for a
>> > >> total hit of about 200MB of bcprov.
>> > >>
>> > >> [1] proposes to make bcprov-jdk15on and bcpkix-jdk15on part of the
>> > >> core provided list of things right along side the
>> > >> logback/slf4j/logging interfaces. This means all things will have
>> > >> access to these going forward and it means it impacts our version
>> > >> compatibility.  Given that it would be a standard/provided thing if we
>> > >> want to upgrade versions or swap it out with something else we could
>> > >> break extensions that then chose to depend on it.  We are not in
>> > >> control of the bcprov api just like we're not in control of the
>> > >> logging APIs.  If there was some important security related fix we
>> > >> needed from bcprov but changing also pulled in api changes for them it
>> > >> could break our extensions.
>> > >>
>> > >> Even with all this said, given the nature, importance, and size
>> > >> benefit, I am in favor of NIFI-2954.  But, would like to highlight
>> > >> this in case others have perspective they'd like to share.
>> > >>
>> > >> [1] https://issues.apache.org/jira/browse/NIFI-2954
>> > >>
>> > >> Thanks
>> > >> Joe
>> > >>
>> >
>>

Re: [DISCUSS] Provided dependency on bcprov-jdk15on

[Michael Moser <mo...@gmail.com>]

I would like to point out that some very common dependencies have been
placed into the nifi-standard-services-api-nar.  This nar is a
Nar-Dependency-Id of several other nars.  Many of these other nars also
include the same dependencies, which would not be used by the nar class
loader.

In particular, nifi-standard-services-api-nar contains nifi-utils,
nifi-processor-utils, and nifi-security-utils.  It also contains the
bcprov-jdk15on and bcpkix-jdk15on jars.

Then looking at nifi-standard-nar, nifi-aws-nar, nifi-kafka-0-10-nar and
more, they also contain these same jars in their bundled-dependencies.
Their Nar-Dependency-Id is nifi-standard-services-api-nar.

-- Mike


On Tue, Nov 1, 2016 at 1:50 PM, Aldrin Piri <al...@gmail.com> wrote:

> I think in light of the fact that the bcprov item was an accidental
> inclusion and not explicitly meant to be handled as such in the framework
> reorganizing seems like the best option as well.  The original intent of
> the PR was to minimize that footprint where we were double dipping given
> the inclusion of nifi-processor-utils.  However, if we manage to shuffle
> items around a bit, the net size reduction could ultimately be the same.
> Most importantly, from a maintenance perspective, bundles will only include
> the library in the NARs that need it which allows that dependency graph to
> be significantly more comprehensible than the scope tweaking originally
> presented.
>
> Will this also work to adjust the dependencies of nifi-processor-utils (and
> potentially nifi-security-utils)?  Currently, the archetype also provides
> nifi-processor-utils, which could lead to similar conflicts for folks that
> were looking to make use of different versions of JARs within a given NAR.
>
> I think the listing Joe provided in the ticket is great and we should be
> mindful of any deltas while performing each release.  For the reasons
> highlighted in this chain and on that issue, one jar inclusion could be
> quite detrimental and cause some difficult issues to those that have
> created extensions.
>
> On Mon, Oct 31, 2016 at 9:39 PM, Joe Witt <jo...@gmail.com> wrote:
>
> > So...I might be a bit of a flip flopper...
> >
> > Had some off-list discussions with a couple people and they really
> > disliked this idea.  They, rightfully, pointed out how this
> > establishes a problematic precedent, how it is really only being done
> > to save space, and that while perhaps not able to pair down to 200MB
> > we can likely do more analysis of the dependency graph to improve the
> > situation.  The real issue that kicked this off was the finding that
> > bcprov (and commons-lang3) had made their way into the root of our
> > classloader (which impacts all classloaders) and this was introduced
> > with the important NIFI-1831 work.  So I've created [1] to address
> > that and return the application classloader to the minimal set of jars
> > necessary which should basically be the runtime application launcher,
> > the logging libraries, and the nifi-api.
> >
> > The ultimate solution to addressing the distribution size concerns
> > will be to address it via the extension registry.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-2971
> >
> > Thanks
> > Joe
> >
> > On Mon, Oct 31, 2016 at 8:51 PM, Andre <an...@fucs.org> wrote:
> > > +1 as well.
> > >
> > > On Tue, Nov 1, 2016 at 2:41 AM, Joe Witt <jo...@gmail.com> wrote:
> > >
> > >> Team,
> > >>
> > >> In the 1.x line bcprov made its way into the root of the classpath/lib
> > >> folder to support the encrypted sensitive properties features.  This
> > >> can be reworked to isolate it a bit more if we need to.
> > >>
> > >> However, in [1] it has been observed that we have bcprov dependencies
> > >> in about 53 places with each instance taking about 4MB of space for a
> > >> total hit of about 200MB of bcprov.
> > >>
> > >> [1] proposes to make bcprov-jdk15on and bcpkix-jdk15on part of the
> > >> core provided list of things right along side the
> > >> logback/slf4j/logging interfaces. This means all things will have
> > >> access to these going forward and it means it impacts our version
> > >> compatibility.  Given that it would be a standard/provided thing if we
> > >> want to upgrade versions or swap it out with something else we could
> > >> break extensions that then chose to depend on it.  We are not in
> > >> control of the bcprov api just like we're not in control of the
> > >> logging APIs.  If there was some important security related fix we
> > >> needed from bcprov but changing also pulled in api changes for them it
> > >> could break our extensions.
> > >>
> > >> Even with all this said, given the nature, importance, and size
> > >> benefit, I am in favor of NIFI-2954.  But, would like to highlight
> > >> this in case others have perspective they'd like to share.
> > >>
> > >> [1] https://issues.apache.org/jira/browse/NIFI-2954
> > >>
> > >> Thanks
> > >> Joe
> > >>
> >
>

Re: [DISCUSS] Provided dependency on bcprov-jdk15on

[Aldrin Piri <al...@gmail.com>]

I think in light of the fact that the bcprov item was an accidental
inclusion and not explicitly meant to be handled as such in the framework
reorganizing seems like the best option as well.  The original intent of
the PR was to minimize that footprint where we were double dipping given
the inclusion of nifi-processor-utils.  However, if we manage to shuffle
items around a bit, the net size reduction could ultimately be the same.
Most importantly, from a maintenance perspective, bundles will only include
the library in the NARs that need it which allows that dependency graph to
be significantly more comprehensible than the scope tweaking originally
presented.

Will this also work to adjust the dependencies of nifi-processor-utils (and
potentially nifi-security-utils)?  Currently, the archetype also provides
nifi-processor-utils, which could lead to similar conflicts for folks that
were looking to make use of different versions of JARs within a given NAR.

I think the listing Joe provided in the ticket is great and we should be
mindful of any deltas while performing each release.  For the reasons
highlighted in this chain and on that issue, one jar inclusion could be
quite detrimental and cause some difficult issues to those that have
created extensions.

On Mon, Oct 31, 2016 at 9:39 PM, Joe Witt <jo...@gmail.com> wrote:

> So...I might be a bit of a flip flopper...
>
> Had some off-list discussions with a couple people and they really
> disliked this idea.  They, rightfully, pointed out how this
> establishes a problematic precedent, how it is really only being done
> to save space, and that while perhaps not able to pair down to 200MB
> we can likely do more analysis of the dependency graph to improve the
> situation.  The real issue that kicked this off was the finding that
> bcprov (and commons-lang3) had made their way into the root of our
> classloader (which impacts all classloaders) and this was introduced
> with the important NIFI-1831 work.  So I've created [1] to address
> that and return the application classloader to the minimal set of jars
> necessary which should basically be the runtime application launcher,
> the logging libraries, and the nifi-api.
>
> The ultimate solution to addressing the distribution size concerns
> will be to address it via the extension registry.
>
> [1] https://issues.apache.org/jira/browse/NIFI-2971
>
> Thanks
> Joe
>
> On Mon, Oct 31, 2016 at 8:51 PM, Andre <an...@fucs.org> wrote:
> > +1 as well.
> >
> > On Tue, Nov 1, 2016 at 2:41 AM, Joe Witt <jo...@gmail.com> wrote:
> >
> >> Team,
> >>
> >> In the 1.x line bcprov made its way into the root of the classpath/lib
> >> folder to support the encrypted sensitive properties features.  This
> >> can be reworked to isolate it a bit more if we need to.
> >>
> >> However, in [1] it has been observed that we have bcprov dependencies
> >> in about 53 places with each instance taking about 4MB of space for a
> >> total hit of about 200MB of bcprov.
> >>
> >> [1] proposes to make bcprov-jdk15on and bcpkix-jdk15on part of the
> >> core provided list of things right along side the
> >> logback/slf4j/logging interfaces. This means all things will have
> >> access to these going forward and it means it impacts our version
> >> compatibility.  Given that it would be a standard/provided thing if we
> >> want to upgrade versions or swap it out with something else we could
> >> break extensions that then chose to depend on it.  We are not in
> >> control of the bcprov api just like we're not in control of the
> >> logging APIs.  If there was some important security related fix we
> >> needed from bcprov but changing also pulled in api changes for them it
> >> could break our extensions.
> >>
> >> Even with all this said, given the nature, importance, and size
> >> benefit, I am in favor of NIFI-2954.  But, would like to highlight
> >> this in case others have perspective they'd like to share.
> >>
> >> [1] https://issues.apache.org/jira/browse/NIFI-2954
> >>
> >> Thanks
> >> Joe
> >>
>

Re: [DISCUSS] Provided dependency on bcprov-jdk15on

[Joe Witt <jo...@gmail.com>]

So...I might be a bit of a flip flopper...

Had some off-list discussions with a couple people and they really
disliked this idea.  They, rightfully, pointed out how this
establishes a problematic precedent, how it is really only being done
to save space, and that while perhaps not able to pair down to 200MB
we can likely do more analysis of the dependency graph to improve the
situation.  The real issue that kicked this off was the finding that
bcprov (and commons-lang3) had made their way into the root of our
classloader (which impacts all classloaders) and this was introduced
with the important NIFI-1831 work.  So I've created [1] to address
that and return the application classloader to the minimal set of jars
necessary which should basically be the runtime application launcher,
the logging libraries, and the nifi-api.

The ultimate solution to addressing the distribution size concerns
will be to address it via the extension registry.

[1] https://issues.apache.org/jira/browse/NIFI-2971

Thanks
Joe

On Mon, Oct 31, 2016 at 8:51 PM, Andre <an...@fucs.org> wrote:
> +1 as well.
>
> On Tue, Nov 1, 2016 at 2:41 AM, Joe Witt <jo...@gmail.com> wrote:
>
>> Team,
>>
>> In the 1.x line bcprov made its way into the root of the classpath/lib
>> folder to support the encrypted sensitive properties features.  This
>> can be reworked to isolate it a bit more if we need to.
>>
>> However, in [1] it has been observed that we have bcprov dependencies
>> in about 53 places with each instance taking about 4MB of space for a
>> total hit of about 200MB of bcprov.
>>
>> [1] proposes to make bcprov-jdk15on and bcpkix-jdk15on part of the
>> core provided list of things right along side the
>> logback/slf4j/logging interfaces. This means all things will have
>> access to these going forward and it means it impacts our version
>> compatibility.  Given that it would be a standard/provided thing if we
>> want to upgrade versions or swap it out with something else we could
>> break extensions that then chose to depend on it.  We are not in
>> control of the bcprov api just like we're not in control of the
>> logging APIs.  If there was some important security related fix we
>> needed from bcprov but changing also pulled in api changes for them it
>> could break our extensions.
>>
>> Even with all this said, given the nature, importance, and size
>> benefit, I am in favor of NIFI-2954.  But, would like to highlight
>> this in case others have perspective they'd like to share.
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-2954
>>
>> Thanks
>> Joe
>>