What do we do with Integrations with no maintainer?

["Wahrmann, Helmut" <he...@rsa.com>]

Hi all, 

What are we doing with integrations having no maintainer?

An example is the morphline sink. It supports Solr 4.3 and Apache has already relased Solr 7.5.0. 
Kite SDK is at 1.1.
Seems that no one is taking care of it.

On the other site we are still "supporting" Elasticsearch 0.90.1, while ElasticSearch is already on 6.5.
Having a dependency of Lucene between Solr and Elastic I cannot push my changes to flume.

So while there would be a maintainer for ElasticSearch, new features cannot be introduced, because of the above.

Shouldn't we deprecate old stuff, where no maintainer is active?
If 1.9 is released with support of outdated SOLR and ElasticSearch, no one would use Flume anyhow.
I doubt that someone would downgrade the ElasticSearch cluster to 0.90 just to be able to pump in events via Flume.

I keep a working ElasticSearch 6.x Fork anyhow, but would like to contribute back to the project.

Note: I have tried to upgrade SOLR support to the newest version, but it fails on the test cases and I don't have anything to test.


Thanks,

Helmut

Re: What do we do with Integrations with no maintainer?

[Ralph Goers <ra...@dslextreme.com>]

Log4j 2 has implemented a plugin system that you could copy.  I will tell you the one obstacle we had to overcome was that locating plugins via ClassLoader scanning at runtime is very slow and was noticeably impacting startup times. That is why Log4j 2 uses an annotation processor at compile time to generate a file containing all the plugins in the jar. Spring handles this a little differently by only scanning packages that have been included in @ComponentScan declarations.

One feature I need to work on for my employer is to have Flume be able to load its configuration from Spring Cloud Config. This would be much easier if Flume could easily be made a Spring boot app (but not necessarily be completely configured via Spring).

Ralph

> On Nov 20, 2018, at 3:36 PM, Bessenyei Balázs Donát <be...@apache.org> wrote:
> 
> I'm in favor of having a plugin system. It would be pretty awesome to
> be able to say in a config that "a1.sources.r1.type =
> com.whatever.awesome-plugin:1.2" and flume or mvn or grape or whatever
> would be able to fetch the deps. It would solve a lot of issues that
> we struggle with right now. - No difficult merges on "but how are we
> going to support this?", instead saying that it's not part of Flume,
> but it integrates well, use at your own risk.
> On the "Basically, flume itself should be just the framework and every
> source/sink/channel/interceptor/etc would come as a plugin." - I think
> there should be _some_ functionality that is supported out of the box
> (hdfs / hbase sink, file / kafka channel, some sources), but plugins
> would be "well supported" and by "well supported" I mean the "how to"
> would be documented, support would be provided to testing (maybe even
> part of the CI pipeline we have now), etc.
> 
> What do you think?
> 
> 
> Donat
> 
> On Mon, Nov 19, 2018 at 11:50 PM Mike Percy <mp...@apache.org> wrote:
>> 
>> I agree with the above... the JAR hell we are currently in seems mostly
>> unresolvable and shedding dependencies seems like a reasonable choice
>> regardless of whether we implement an isolated classloader or not.
>> 
>> Mike
>> 
>> On Mon, Nov 19, 2018 at 6:02 AM Ferenc Szabo <sz...@apache.org> wrote:
>> 
>>> Hi all,
>>> 
>>> I somewhat agree with Helmut.
>>> I think we should detach every component from the framework and provide an
>>> isolated classloader for them to avoid the dependency issues we have now.
>>> 
>>> Basically, flume itself should be just the framework and every
>>> source/sink/channel/interceptor/etc would come as a plugin.
>>> We would have "official" plugins in the flume repository but as separate
>>> maven projects.
>>> 
>>> After the 1.9 release, we could start planning it for 2.0 as it would be
>>> easiest (or only possible) to do with breaking changes, however, it would
>>> be good to aim for compatibility with 1.x plugins.
>>> 
>>> What do you think about that?
>>> 
>>> Additionally, I would be happy to maintain a plugin list for flume with the
>>> 3rd party/community plugins.
>>> 
>>> Regards,
>>> 
>>> Ferenc
>>> 
>>> 
>>> On Mon, Nov 19, 2018 at 1:19 PM Wahrmann, Helmut <he...@rsa.com>
>>> wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> What are we doing with integrations having no maintainer?
>>>> 
>>>> An example is the morphline sink. It supports Solr 4.3 and Apache has
>>>> already relased Solr 7.5.0.
>>>> Kite SDK is at 1.1.
>>>> Seems that no one is taking care of it.
>>>> 
>>>> On the other site we are still "supporting" Elasticsearch 0.90.1, while
>>>> ElasticSearch is already on 6.5.
>>>> Having a dependency of Lucene between Solr and Elastic I cannot push my
>>>> changes to flume.
>>>> 
>>>> So while there would be a maintainer for ElasticSearch, new features
>>>> cannot be introduced, because of the above.
>>>> 
>>>> Shouldn't we deprecate old stuff, where no maintainer is active?
>>>> If 1.9 is released with support of outdated SOLR and ElasticSearch, no
>>> one
>>>> would use Flume anyhow.
>>>> I doubt that someone would downgrade the ElasticSearch cluster to 0.90
>>>> just to be able to pump in events via Flume.
>>>> 
>>>> I keep a working ElasticSearch 6.x Fork anyhow, but would like to
>>>> contribute back to the project.
>>>> 
>>>> Note: I have tried to upgrade SOLR support to the newest version, but it
>>>> fails on the test cases and I don't have anything to test.
>>>> 
>>>> 
>>>> Thanks,
>>>> 
>>>> Helmut
>>>> 
>>>> 
>>> 
> 

Re: What do we do with Integrations with no maintainer?

[Bessenyei Balázs Donát <be...@apache.org>]

I'm in favor of having a plugin system. It would be pretty awesome to
be able to say in a config that "a1.sources.r1.type =
com.whatever.awesome-plugin:1.2" and flume or mvn or grape or whatever
would be able to fetch the deps. It would solve a lot of issues that
we struggle with right now. - No difficult merges on "but how are we
going to support this?", instead saying that it's not part of Flume,
but it integrates well, use at your own risk.
On the "Basically, flume itself should be just the framework and every
source/sink/channel/interceptor/etc would come as a plugin." - I think
there should be _some_ functionality that is supported out of the box
(hdfs / hbase sink, file / kafka channel, some sources), but plugins
would be "well supported" and by "well supported" I mean the "how to"
would be documented, support would be provided to testing (maybe even
part of the CI pipeline we have now), etc.

What do you think?


Donat

On Mon, Nov 19, 2018 at 11:50 PM Mike Percy <mp...@apache.org> wrote:
>
> I agree with the above... the JAR hell we are currently in seems mostly
> unresolvable and shedding dependencies seems like a reasonable choice
> regardless of whether we implement an isolated classloader or not.
>
> Mike
>
> On Mon, Nov 19, 2018 at 6:02 AM Ferenc Szabo <sz...@apache.org> wrote:
>
> > Hi all,
> >
> > I somewhat agree with Helmut.
> > I think we should detach every component from the framework and provide an
> > isolated classloader for them to avoid the dependency issues we have now.
> >
> > Basically, flume itself should be just the framework and every
> > source/sink/channel/interceptor/etc would come as a plugin.
> > We would have "official" plugins in the flume repository but as separate
> > maven projects.
> >
> > After the 1.9 release, we could start planning it for 2.0 as it would be
> > easiest (or only possible) to do with breaking changes, however, it would
> > be good to aim for compatibility with 1.x plugins.
> >
> >  What do you think about that?
> >
> > Additionally, I would be happy to maintain a plugin list for flume with the
> > 3rd party/community plugins.
> >
> > Regards,
> >
> > Ferenc
> >
> >
> > On Mon, Nov 19, 2018 at 1:19 PM Wahrmann, Helmut <he...@rsa.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > What are we doing with integrations having no maintainer?
> > >
> > > An example is the morphline sink. It supports Solr 4.3 and Apache has
> > > already relased Solr 7.5.0.
> > > Kite SDK is at 1.1.
> > > Seems that no one is taking care of it.
> > >
> > > On the other site we are still "supporting" Elasticsearch 0.90.1, while
> > > ElasticSearch is already on 6.5.
> > > Having a dependency of Lucene between Solr and Elastic I cannot push my
> > > changes to flume.
> > >
> > > So while there would be a maintainer for ElasticSearch, new features
> > > cannot be introduced, because of the above.
> > >
> > > Shouldn't we deprecate old stuff, where no maintainer is active?
> > > If 1.9 is released with support of outdated SOLR and ElasticSearch, no
> > one
> > > would use Flume anyhow.
> > > I doubt that someone would downgrade the ElasticSearch cluster to 0.90
> > > just to be able to pump in events via Flume.
> > >
> > > I keep a working ElasticSearch 6.x Fork anyhow, but would like to
> > > contribute back to the project.
> > >
> > > Note: I have tried to upgrade SOLR support to the newest version, but it
> > > fails on the test cases and I don't have anything to test.
> > >
> > >
> > > Thanks,
> > >
> > > Helmut
> > >
> > >
> >

Re: What do we do with Integrations with no maintainer?

[Mike Percy <mp...@apache.org>]

I agree with the above... the JAR hell we are currently in seems mostly
unresolvable and shedding dependencies seems like a reasonable choice
regardless of whether we implement an isolated classloader or not.

Mike

On Mon, Nov 19, 2018 at 6:02 AM Ferenc Szabo <sz...@apache.org> wrote:

> Hi all,
>
> I somewhat agree with Helmut.
> I think we should detach every component from the framework and provide an
> isolated classloader for them to avoid the dependency issues we have now.
>
> Basically, flume itself should be just the framework and every
> source/sink/channel/interceptor/etc would come as a plugin.
> We would have "official" plugins in the flume repository but as separate
> maven projects.
>
> After the 1.9 release, we could start planning it for 2.0 as it would be
> easiest (or only possible) to do with breaking changes, however, it would
> be good to aim for compatibility with 1.x plugins.
>
>  What do you think about that?
>
> Additionally, I would be happy to maintain a plugin list for flume with the
> 3rd party/community plugins.
>
> Regards,
>
> Ferenc
>
>
> On Mon, Nov 19, 2018 at 1:19 PM Wahrmann, Helmut <he...@rsa.com>
> wrote:
>
> > Hi all,
> >
> > What are we doing with integrations having no maintainer?
> >
> > An example is the morphline sink. It supports Solr 4.3 and Apache has
> > already relased Solr 7.5.0.
> > Kite SDK is at 1.1.
> > Seems that no one is taking care of it.
> >
> > On the other site we are still "supporting" Elasticsearch 0.90.1, while
> > ElasticSearch is already on 6.5.
> > Having a dependency of Lucene between Solr and Elastic I cannot push my
> > changes to flume.
> >
> > So while there would be a maintainer for ElasticSearch, new features
> > cannot be introduced, because of the above.
> >
> > Shouldn't we deprecate old stuff, where no maintainer is active?
> > If 1.9 is released with support of outdated SOLR and ElasticSearch, no
> one
> > would use Flume anyhow.
> > I doubt that someone would downgrade the ElasticSearch cluster to 0.90
> > just to be able to pump in events via Flume.
> >
> > I keep a working ElasticSearch 6.x Fork anyhow, but would like to
> > contribute back to the project.
> >
> > Note: I have tried to upgrade SOLR support to the newest version, but it
> > fails on the test cases and I don't have anything to test.
> >
> >
> > Thanks,
> >
> > Helmut
> >
> >
>

Re: What do we do with Integrations with no maintainer?

[Ferenc Szabo <sz...@apache.org>]

Hi all,

I somewhat agree with Helmut.
I think we should detach every component from the framework and provide an
isolated classloader for them to avoid the dependency issues we have now.

Basically, flume itself should be just the framework and every
source/sink/channel/interceptor/etc would come as a plugin.
We would have "official" plugins in the flume repository but as separate
maven projects.

After the 1.9 release, we could start planning it for 2.0 as it would be
easiest (or only possible) to do with breaking changes, however, it would
be good to aim for compatibility with 1.x plugins.

 What do you think about that?

Additionally, I would be happy to maintain a plugin list for flume with the
3rd party/community plugins.

Regards,

Ferenc


On Mon, Nov 19, 2018 at 1:19 PM Wahrmann, Helmut <he...@rsa.com>
wrote:

> Hi all,
>
> What are we doing with integrations having no maintainer?
>
> An example is the morphline sink. It supports Solr 4.3 and Apache has
> already relased Solr 7.5.0.
> Kite SDK is at 1.1.
> Seems that no one is taking care of it.
>
> On the other site we are still "supporting" Elasticsearch 0.90.1, while
> ElasticSearch is already on 6.5.
> Having a dependency of Lucene between Solr and Elastic I cannot push my
> changes to flume.
>
> So while there would be a maintainer for ElasticSearch, new features
> cannot be introduced, because of the above.
>
> Shouldn't we deprecate old stuff, where no maintainer is active?
> If 1.9 is released with support of outdated SOLR and ElasticSearch, no one
> would use Flume anyhow.
> I doubt that someone would downgrade the ElasticSearch cluster to 0.90
> just to be able to pump in events via Flume.
>
> I keep a working ElasticSearch 6.x Fork anyhow, but would like to
> contribute back to the project.
>
> Note: I have tried to upgrade SOLR support to the newest version, but it
> fails on the test cases and I don't have anything to test.
>
>
> Thanks,
>
> Helmut
>
>