You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "Peter Wicks (pwicks)" <pw...@micron.com> on 2019/08/22 16:19:29 UTC
RE: [EXT] Re: [DISCUSS] Assembly size for 1.10
This sounds quite reasonable to me. Lets keep the big changes for the next version and move forward with just excluding a few NAR's from the convenience assembly for 1.10.
-----Original Message-----
From: Aldrin Piri <al...@gmail.com>
Sent: Thursday, August 22, 2019 7:33 AM
To: dev <de...@nifi.apache.org>
Subject: [EXT] Re: [DISCUSS] Assembly size for 1.10
Given the time needed to do the other options "right," the removal of the optional NARs seems most prudent to avoid making the release of 1.10 unduly protracted.
The listing above seems quite reasonable.
We could still have these items be built but just not included in the resulting assembly. This would allow excluded NAR artifacts to make their way to/be accessible from repositories (e.g. nifi-media-nar:
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsearch.maven.org%2Fartifact%2Forg.apache.nifi%2Fnifi-media-nar%2F1.9.2%2Fnar&data=02%7C01%7Cpwicks%40micron.com%7Cef540917424f44d6f34008d727056b1f%7Cf38a5ecd28134862b11bac1d563c806f%7C0%7C0%7C637020776542195375&sdata=uCTRdpDRs0J393FvZoyZ9M3b8PV8NR%2F8Wai5ahG9W9w%3D&reserved=0)
. Accordingly, with some notes on migration, it would still be possible for users to customize their distribution with a few curl/wget commands and not be required to build anything. This is likely the experience that extension registry will eventually provide in a more user friendly/integrated manner but helps us bridge the gap in the interim.
On Thu, Aug 22, 2019 at 9:15 AM Bryan Bende <bb...@gmail.com> wrote:
> I think putting all the NARs in an extension registry where users can
> pick and choose what they want will probably achieve the same thing.
> Either of those options require hosting binaries somewhere and making
> them accessible to be downloaded into a base installation.
>
> For the short term, I think we should re-focus on which NARs to make
> optional for 1.10.0. So far proposed options are...
>
> - nifi-flume-nar (32 MB)
> - nifi-other-graph-services-nar (40 MB)
> - nifi-kafka-0-8-nar (21 MB)
> - nifi-media-nar (63 MB)
>
> That would give us roughly 150 MB back.
>
> If we want to keep going, I'm wondering if the Druid NAR would be
> another candidate (43 MB).
>
> On Thu, Aug 22, 2019 at 4:32 AM Edward Armes <ed...@gmail.com>
> wrote:
> >
> > Could we potentially look at removing the embedded external
> > dependences
> for
> > the nars and have them resolve on first start up using an embed
> > manger
> like
> > what spark jobs and some of the OSGI frameworks do?
> >
> > Edward
> >
> > On Thu, 22 Aug 2019, 04:48 Bryan Bende, <bb...@gmail.com> wrote:
> >
> > > I would vote for doing the reorganization after this release
> > > because
> on its
> > > own it’s doesn’t really solver this specific problem. We would be
> moving
> > > all the NARs to another git repo, but still need a way to
> > > distribute
> them.
> > > Some of the options would be...
> > >
> > > A) Pick and choose only some of them to be included in the
> > > standard distribution (same thing we are doing now and doesn’t
> > > require reorganization).
> > >
> > > B) Create a couple of different NAR assemblies that each on their
> > > own
> don’t
> > > exceed the size limit, and could easily be dropped into the base
> > > distribution (In previous discussions no one could agree on how to
> > > do this).
> > >
> > > C) Release each NAR independently to a central extension registry
> > > to be easily installed to anyone’s NiFi (ultimate goal but doesn’t
> > > exist
> yet).
> > >
> > > On Wed, Aug 21, 2019 at 10:13 PM Jeff <jt...@gmail.com> wrote:
> > >
> > > > Realistically we'll have to do the reorganization and removal of
> optional
> > > > NARs. Regardless of the reorganization, the convenience binary
> > > > has
> grown
> > > > too large. Given that, it probably makes the most sense to pull
> > > > out
> an
> > > > initial set of optional NARs from the convenience binary this
> release,
> > > and
> > > > implement the reorganization in the following release.
> > > >
> > > > On Wed, Aug 21, 2019 at 10:06 PM Mike Thomsen <
> mikerthomsen@gmail.com>
> > > > wrote:
> > > >
> > > > > My impression was that the reorganization initiative has
> > > > > enough
> steps
> > > > that
> > > > > it might actually be wise for us to at least take on a chunk
> > > > > of
> them
> > > now
> > > > so
> > > > > we don't end up with a release that suddenly feels to NiFi
> > > > > users
> the
> > > way
> > > > > Java 9 felt to Java 7 and 8 users.
> > > > >
> > > > > On Wed, Aug 21, 2019 at 9:59 PM Jeff <jt...@gmail.com> wrote:
> > > > >
> > > > > > How motivated could we be to do the reorganization of the
> > > > > > NiFi
> > > > repository
> > > > > > before the 1.10.0 release? It sounds like we have a few
> > > > > > paths
> to get
> > > > the
> > > > > > resulting convenience binary down below the size limit. If
> > > > > > we
> don't
> > > do
> > > > > the
> > > > > > reorganization for this upcoming release, we should make it
> > > > > > a top
> > > > > priority
> > > > > > for the following release.
> > > > > >
> > > > > > On Wed, Aug 21, 2019 at 6:22 PM Mike Thomsen <
> mikerthomsen@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Another factor on why that would be a good idea: I might
> > > > > > > soon
> have
> > > to
> > > > > > pivot
> > > > > > > and do the R&D on adding dgraph support to graph bundle.
> > > > > > > So
> it's
> > > not
> > > > > > > altogether unlikely that it might need to be refactored to
> > > > > > > make
> > > room
> > > > > for
> > > > > > > other graph tech.
> > > > > > >
> > > > > > > On Wed, Aug 21, 2019 at 6:20 PM Mike Thomsen <
> > > mikerthomsen@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Go ahead and remove the whole graph bundle from the
> assembly. I
> > > > would
> > > > > > > > recommend cutting a release of it separately and putting
> > > > > > > > it
> up on
> > > > > > > GitHub's
> > > > > > > > releases listing if that's a possibility for us/INFRA w/
> GitHub.
> > > > Most
> > > > > > of
> > > > > > > > our potential graph users are savvy enough that if add a
> > > > > > > > few
> > > > steps, I
> > > > > > > don't
> > > > > > > > see it causing any grief on them getting it stood up and
> giving
> > > us
> > > > > > > feedback.
> > > > > > > >
> > > > > > > > Might be a good idea also to add a "full-build" profile
> > > > > > > > to
> the
> > > > > assembly
> > > > > > > so
> > > > > > > > that we can throw the whole kitchen sink into an
> > > > > > > > unofficial
> build
> > > > if
> > > > > we
> > > > > > > > build it ourselves for someone else.
> > > > > > > >
> > > > > > > > On Wed, Aug 21, 2019 at 3:09 PM Joe Witt
> > > > > > > > <joe.witt@gmail.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > >> Bryan
> > > > > > > >>
> > > > > > > >> I agree with all of that. What does that get us to?
> > > > > > > >>
> > > > > > > >> Thanks
> > > > > > > >>
> > > > > > > >> On Wed, Aug 21, 2019 at 3:03 PM Bryan Bende <
> bbende@gmail.com>
> > > > > wrote:
> > > > > > > >>
> > > > > > > >> > I would vote to make nifi-flume-nar optional, and it
> > > > > > > >> > looks
> > > like
> > > > > > > >> > nifi-other-graph-services-nar might be new since last
> release,
> > > > so
> > > > > > > >> > since that is in the top 10 and not released yet, it
> > > > > > > >> > might
> > > also
> > > > > be a
> > > > > > > >> > good candidate (not downplaying the usefulness of
> anything in
> > > > that
> > > > > > > >> > NAR).
> > > > > > > >> >
> > > > > > > >> > I would also think we could consider the
> nifi-kafka-0-8-nar
> > > > since
> > > > > > > >> > Kafka 0.8 is quite old at this point, and we already
> > > > > > > >> > have
> > > other
> > > > > > Kafka
> > > > > > > >> > NARs for 0.9, 0.10, 0.11, 1.0, and 2.0. Might even
> consider
> > > > > > dropping a
> > > > > > > >> > few more versions from default assembly.
> > > > > > > >> >
> > > > > > > >> > On Wed, Aug 21, 2019 at 2:45 PM Aldrin Piri <
> > > aldrin@apache.org>
> > > > > > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > Hi folks,
> > > > > > > >> > >
> > > > > > > >> > > Doing a recent PR review and build, it seems that
> master has
> > > > > > amassed
> > > > > > > >> some
> > > > > > > >> > > additional size since our 1.9.2 release approaching
> 200MB.
> > > > > > > >> > >
> > > > > > > >> > > Unfortunately, this is problematic and needs to be
> addressed
> > > > in
> > > > > > > >> advance
> > > > > > > >> > of
> > > > > > > >> > > our 1.10 release. INFRA has been more than helpful
> making
> > > one
> > > > > off
> > > > > > > >> > > exceptions [1][2] for the larger assembly to get
> published
> > > to
> > > > > the
> > > > > > > ASF
> > > > > > > >> > > repository and its associated mirrors, but another
> release
> > > > that
> > > > > is
> > > > > > > >> even
> > > > > > > >> > > larger is not something we can allow. In a Linux
> > > environment,
> > > > > the
> > > > > > > >> master
> > > > > > > >> > > build reports in at 1575671276 which puts us over
> > > > > > > >> > > the
> hard
> > > > limit
> > > > > > > >> > > highlighted in [2].
> > > > > > > >> > >
> > > > > > > >> > > We had a prior community discussion [3] about
> > > > > > > >> > > splitting
> the
> > > > > > > framework
> > > > > > > >> and
> > > > > > > >> > > extension repos and I am hoping to revive that
> discussion,
> > > in
> > > > > > part.
> > > > > > > >> We
> > > > > > > >> > > certainly know what our longer term goals and
> > > > > > > >> > > ambitions
> are
> > > > but
> > > > > > > need a
> > > > > > > >> > fix
> > > > > > > >> > > in the interim. In the current state, we will not
> > > > > > > >> > > be
> able
> > > to
> > > > > make
> > > > > > > our
> > > > > > > >> > > convenience binaries available at the conclusion of
> > > > > > > >> > > the
> > > > release
> > > > > > > >> process.
> > > > > > > >> > >
> > > > > > > >> > > At minimum we should evaluate which bundles are
> eligible to
> > > > get
> > > > > > > >> treated
> > > > > > > >> > as
> > > > > > > >> > > optional dependencies and only enabled via profile,
> > > > > > > >> > > much
> > > like
> > > > > the
> > > > > > > work
> > > > > > > >> > that
> > > > > > > >> > > has occurred surrounding some of our other, hefty NARs.
> [4]
> > > A
> > > > > > > listing
> > > > > > > >> of
> > > > > > > >> > > the top 50 largest NARs, excluding framework and
> standard,
> > > is
> > > > > > > >> available
> > > > > > > >> > in
> > > > > > > >> > > a gist [5]. The nifi-media-nar looks to be a good
> initial
> > > > > > candidate
> > > > > > > >> for
> > > > > > > >> > > exclusion.
> > > > > > > >> > >
> > > > > > > >> > > Thanks for your consideration!
> > > > > > > >> > >
> > > > > > > >> > > --aldrin
> > > > > > > >> > >
> > > > > > > >> > > [1]
> > > > > > > >> > > https://nam01.safelinks.protection.outlook.com/?url
> > > > > > > >> > > =https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2F
> > > > > > > >> > > INFRA-11252&data=02%7C01%7Cpwicks%40micron.com%
> > > > > > > >> > > 7Cef540917424f44d6f34008d727056b1f%7Cf38a5ecd281348
> > > > > > > >> > > 62b11bac1d563c806f%7C0%7C0%7C637020776542195375&
> > > > > > > >> > > ;sdata=XHNH6XIP5MOoSNRx2m%2F7QTka7XRKpB2Nry3lmPqigl
> > > > > > > >> > > M%3D&reserved=0 [2]
> > > > > > > >> > > https://nam01.safelinks.protection.outlook.com/?url
> > > > > > > >> > > =https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2F
> > > > > > > >> > > INFRA-15816&data=02%7C01%7Cpwicks%40micron.com%
> > > > > > > >> > > 7Cef540917424f44d6f34008d727056b1f%7Cf38a5ecd281348
> > > > > > > >> > > 62b11bac1d563c806f%7C0%7C0%7C637020776542195375&
> > > > > > > >> > > ;sdata=5eoIdjF5BVlZzqk6CewcCQTDdGopQtr5soPysq1Ykj4%
> > > > > > > >> > > 3D&reserved=0
> > > > > > > >> > > [3]
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> s.apache.org%2Fthread.html%2F939a7630a2e32594cd10444e48b7a1321fd9ce518
> 34d911a8c04b6a9%40&data=02%7C01%7Cpwicks%40micron.com%7Cef54091742
> 4f44d6f34008d727056b1f%7Cf38a5ecd28134862b11bac1d563c806f%7C0%7C0%7C63
> 7020776542195375&sdata=r5XRcoDvs28K69wKsiDjQFx%2FaCSwxxpyNMiupuRxZ
> IY%3D&reserved=0
> > > > > > > >> > <dev.nifi.apache.org>
> > > > > > > >> > > [4]
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fapache%2Fnifi%2Fblob%2Fmaster%2Fnifi-assembly%2Fpom.xml%23L80
> 7-L875&data=02%7C01%7Cpwicks%40micron.com%7Cef540917424f44d6f34008
> d727056b1f%7Cf38a5ecd28134862b11bac1d563c806f%7C0%7C0%7C63702077654219
> 5375&sdata=%2B8STidbyjnF8j0KFau0I3sPmZZsD2BXPJbbkRNiNtfE%3D&re
> served=0
> > > > > > > >> > > [5]
> > > > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A
> > > > > > %2F%2Fgist.github.com%2Fapiri%2F4d9a02f9f6b46867b601956df83b
> > > > > > 6d8c&data=02%7C01%7Cpwicks%40micron.com%7Cef540917424f44
> > > > > > d6f34008d727056b1f%7Cf38a5ecd28134862b11bac1d563c806f%7C0%7C
> > > > > > 0%7C637020776542195375&sdata=McdNKumvtuT1Gw7nIfvIwSQoDZH
> > > > > > 31RjS6ZvrNEX7GlA%3D&reserved=0
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > --
> > > Sent from Gmail Mobile
> > >
>