You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Wahrmann, Helmut" <he...@rsa.com> on 2018/01/23 08:57:41 UTC
Merging to trunk?
Hi,
Who decides if/when a patch or pull request gets merged to trunk?
Reason I am asking is for the Elasticsearch support. The current code in trunk does not work with ES > 2.x.
Currently Elasticsearch is at 6.1.
In FLUME-3021 we have several patches since March last year.
I have a patched Flume running at a customer since April last year without any problems.
So why not merging those changes into the trunk?
As I stated in my comment in FLUME-3021, it will not cause any problems, cause the current trunk won't work with newer ES versions anyhow.
I doubt that someone is out there still running ES 0.95.
best regards,
Helmut
RE: Merging to trunk?
Posted by "Wahrmann, Helmut" <he...@rsa.com>.
Hi Mike,
I gave it a try and changed the Solr version to 7.2.1. It all compiled well.
Only the tests were failing, because SolrServer, which is needed in the test cases to connect to the embedded Test Server is no longer available in org.apache.solr.client.solrj.
But that's not needed in the "production" version.
I cannot test however if it really works, since I don't have access to a Solr installation.
I understand your discussion with the dependency problem, but I need to move ahead.
It doesn't help me to have a Flume 1.9 or 2.0 released with no dependency problems, but it doesn't support the latest Elasticsearch client, which are needed by my customer.
And by the time, you will release Flume 1.9, the ES project is at version 8, where they will drop support for the Transport client.
So I am in the urgent need to develop support for the new ElasticSearch Rest Client.
For that I need the changes which are done in Flume-3021.
So I continue my work in a fork and will publish Pull requests . Once the dependencies problems are sorted out, you're welcome to include my changes in your trunk.
best regards,
Helmut
-----Original Message-----
From: Mike Percy [mailto:mpercy@apache.org]
Sent: Dienstag, 30. Jänner 2018 04:57
To: dev@flume.apache.org
Subject: Re: Merging to trunk?
On Mon, Jan 29, 2018 at 10:48 AM, Wahrmann, Helmut <helmut.wahrmann@rsa.com
> wrote:
>
> I think it is not a problem to distribute the modules together if we
> have maintainers for them.
> It doesn't make sense to release Flume 1.9 in 2018 with support for
> module versions which were current in 2013.
> Best way would be to identify people willing to upgrade the modules to
> their latest version.
> If we can't find someone, we should distribute it separately.
I would be OK with upgrading Solr and ES simultaneously if we can do it in a compatible way. However, there is no guarantee that the latest versions of Solr and ES will be dependency-compatible. If they are, I would certainly be in favor of merging such a patch.
On Mon, Jan 29, 2018 at 2:49 AM, Ferenc Szabo <sz...@apache.org> wrote:
> I believe, that a clean and maintainable solution would be if the
> engine/framework itself could be separated from everything else, the
> dependency directions would be fixed, plugins would depend only on
> api-s and after that, any source/sink/etc implementation would come
> as an individual module/plugin and would be loaded with an isolated classloader.
>
I agree that isolating Flume plugins from each other would solve this problem once and for all. I wonder if we can do both things: come up with a short-term "band-aid" patch to allow us to upgrade ES / Solr and also come up with a longer-term plan to solve the underlying problem.
Mike
Re: Merging to trunk?
Posted by Mike Percy <mp...@apache.org>.
On Mon, Jan 29, 2018 at 10:48 AM, Wahrmann, Helmut <helmut.wahrmann@rsa.com
> wrote:
>
> I think it is not a problem to distribute the modules together if we have
> maintainers for them.
> It doesn't make sense to release Flume 1.9 in 2018 with support for module
> versions which were current in 2013.
> Best way would be to identify people willing to upgrade the modules to
> their latest version.
> If we can't find someone, we should distribute it separately.
I would be OK with upgrading Solr and ES simultaneously if we can do it in
a compatible way. However, there is no guarantee that the latest versions
of Solr and ES will be dependency-compatible. If they are, I would
certainly be in favor of merging such a patch.
On Mon, Jan 29, 2018 at 2:49 AM, Ferenc Szabo <sz...@apache.org> wrote:
> I believe, that a clean and maintainable solution would be if the
> engine/framework itself could be separated from everything else,
> the dependency directions would be fixed, plugins would depend only on
> api-s and after that, any source/sink/etc implementation would come as an
> individual module/plugin and would be loaded with an isolated classloader.
>
I agree that isolating Flume plugins from each other would solve this
problem once and for all. I wonder if we can do both things: come up with a
short-term "band-aid" patch to allow us to upgrade ES / Solr and also come
up with a longer-term plan to solve the underlying problem.
Mike
Re: Merging to trunk?
Posted by Ferenc Szabo <sz...@apache.org>.
I believe, that a clean and maintainable solution would be if the
engine/framework itself could be separated from everything else,
the dependency directions would be fixed, plugins would depend only on
api-s and after that, any source/sink/etc implementation would come as an
individual module/plugin and would be loaded with an isolated classloader.
This way anyone could implement anything with any version. It would be
possible to use a different version of ES at the same time if you need it.
On Mon, Jan 29, 2018 at 10:48 AM, Wahrmann, Helmut <he...@rsa.com>
wrote:
> I think it is not a problem to distribute the modules together if we have
> maintainers for them.
> It doesn't make sense to release Flume 1.9 in 2018 with support for module
> versions which were current in 2013.
>
> Best way would be to identify people willing to upgrade the modules to
> their latest version.
> If we can't find someone, we should distribute it separately.
>
> best regards,
>
> Helmut
>
>
> -----Original Message-----
> From: Mike Percy [mailto:mpercy@apache.org]
> Sent: Montag, 29. Jänner 2018 09:12
> To: dev@flume.apache.org
> Subject: Re: Merging to trunk?
>
> Interesting proposal Yonghao.
>
> I think it's worth pointing out that today, a Flume build will work "out
> of the box" with any combination of supported plugins since they are all
> already in the Flume classpath. The downside of that, of course, is that
> the dependency hell makes it very hard to upgrade modules. The benefit is
> that it's nice from a usability perspective.
>
> However, if we are willing to drop this "out of the box" usage feature,
> then we could allow for separate modules to have conflicting dependencies
> (i.e. Solr and ElasticSearch) as long as they are not loaded in the same
> Flume agent.
>
> What do you folks think about removing the Solr Sink and the ElasticSearch
> Sink from the default distribution and instead distribute them as separate
> (fat) jars? If we do it for those two then it might also make sense to do
> it for other modules as well.
>
> I am not sure this is very user friendly in the common case where people
> just want to use the HDFS and HBase sink out of the box. So maybe we leave
> those as part of the "default" module setup?
>
> Obviously, changing how modules get included into the classpath would be a
> breaking change. I think a change that drastic would call for a Flume 2.0.0
> release in order to indicate that it's not backwards compatible.
>
> Thoughts?
>
> Mike
>
> On Sat, Jan 27, 2018 at 10:50 PM, Yonghao Zou <yo...@gmail.com>
> wrote:
>
> > Maybe the Maven shade plugin can solve this, we can package each
> > module to a fat jar and they will not depend on other modules.
> >
> > 2018-01-27 6:14 GMT+08:00 Wahrmann, Helmut <he...@rsa.com>:
> >
> > > fully agreed.
> > > But we shouldn't do that do support a 4 years old component.
> > > The goal should be to support the latest releases.
> > > It is ridiculous to release flume 1.9 with elasticsearch 0.90.1
> > > support, when 6.x is the current version.
> > >
> > >
> > >
> > > Best regards,
> > >
> > > Helmut Wahrmann
> > >
> > >
> > > -------- Ursprüngliche Nachricht --------
> > > Von: Ralph Goers <ra...@dslextreme.com>
> > > Datum: 26.01.18 22:12 (GMT+01:00)
> > > An: dev@flume.apache.org
> > > Betreff: Re: Merging to trunk?
> > >
> > > The “right” way to deal with this from a Maven perspective is to
> > > declare the version you want in the dependencyManagement section of
> > > the parent
> > pom
> > > and to not specify any versions in child poms. Then use the Maven
> > enforcer
> > > plugin to make sure everything required has a declaration in the
> > > dependencyManagement section.
> > >
> > > Ralph
> > >
> > > > On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut
> > > > <helmut.wahrmann@rsa.com
> > >
> > > wrote:
> > > >
> > > > Hi Mike,
> > > >
> > > > This only shows that some other component needs to be updated as
> well.
> > > > Which software relies on Lucene 4.3.0, which was released in 2013?
> > > > It is most probably not used by anyone else anymore.
> > > >
> > > > This needs to be solved. It makes no sense to release Flume 1.9
> > > > with
> > > support for ElasticSearch 0.90.1.
> > > > And it doesn't make sense either to release Flume 1.9 with support
> > > > for
> > a
> > > component that relies on Lucene 4.3.0.
> > > >
> > > > We will always have dependency problems, if one component is
> > > > updated to
> > > a newer version and others not.
> > > >
> > > > To get rid of the conflict is probably to set "optional" to true
> > > > in the
> > > pom.xml.
> > > > Then we don't distribute those Jars in the flume lib and the users
> > > > need
> > > to set their class path to point e.g. to Elasticsearch and then the
> > correct
> > > libs will be used.
> > > > I doubt that a Elasticsearch user will also use some other
> > > > component
> > > with e.g. this outdated Lucene.
> > > >
> > > > best regards,
> > > >
> > > > Helmut
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Mike Percy [mailto:mpercy@apache.org]
> > > > Sent: Donnerstag, 25. Jänner 2018 23:01
> > > > To: dev@flume.apache.org
> > > > Subject: Re: Merging to trunk?
> > > >
> > > > Hi Helmut,
> > > > I wrote a small Perl script to identify dependency conflicts based
> > > > on
> > > output from mvn dependency:tree and posted it here:
> > > https://gist.github.com/
> > > > mpercy/39614d770864bdd0c386befd5e8a1840
> > > >
> > > > I ran that on the current trunk and it actually found some errors
> > > > which
> > > should be fixed:
> > > >
> > > > Version conflict: package com.google.guava:guava:jar needed in 2
> > > different
> > > > versions: (11.0.2, 18.0)
> > > > Version conflict: package
> > > > commons-httpclient:commons-httpclient:jar
> > > needed in 2 different versions: (3.0.1, 3.1) Version conflict:
> > > package commons-logging:commons-logging:jar needed in 2 different
> versions:
> > > (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar
> > needed
> > > in 2 different versions: (1.5.4, 1.6) Version conflict: package
> > > net.sf.jopt-simple:jopt-simple:jar needed in 2 different versions:
> > > (3.2,
> > > 4.7) Version conflict: package
> > > org.codehaus.jackson:jackson-jaxrs:jar
> > > needed in
> > > > 2 different versions: (1.8.3, 1.8.8)
> > > > ERROR: 6 package version conflicts identified
> > > >
> > > > However, after applying your patch on top of trunk and running it
> > again,
> > > it identified new conflicts:
> > > >
> > > > Version conflict: package com.google.guava:guava:jar needed in 2
> > > different
> > > > versions: (11.0.2, 18.0)
> > > > Version conflict: package
> > > > commons-httpclient:commons-httpclient:jar
> > > needed in 2 different versions: (3.0.1, 3.1) Version conflict:
> > > package commons-logging:commons-logging:jar needed in 2 different
> versions:
> > > (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar
> > needed
> > > in 2 different versions: (1.5.4, 1.6) Version conflict: package
> > > net.sf.jopt-simple:jopt-simple:jar needed in 3 different versions:
> > > (3.2, 4.7, 5.0.2) Version conflict: package
> > > org.apache.lucene:lucene- analyzers-common:jar
> > > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict:
> > package
> > > org.apache.lucene:lucene-core:jar needed in 2 different versions:
> > (4.3.0,
> > > 7.1.0) Version conflict: package
> > > org.apache.lucene:lucene-grouping:jar
> > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict:
> > > package org.apache.lucene:lucene-highlighter:jar needed in 2
> different versions:
> > > (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-
> > memory:jar
> > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict:
> > > package org.apache.lucene:lucene-misc:jar needed in 2 different
> versions:
> > (4.3.0,
> > > 7.1.0) Version conflict: package
> > > org.apache.lucene:lucene-queries:jar
> > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict:
> > > package org.apache.lucene:lucene-queryparser:jar needed in 2
> different versions:
> > > (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-
> > spatial:jar
> > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict:
> > > package org.apache.lucene:lucene-suggest:jar needed in 2 different
> versions:
> > > (4.3.0, 7.1.0) Version conflict: package
> > > org.codehaus.jackson:jackson-
> > jaxrs:jar
> > > needed in
> > > > 2 different versions: (1.8.3, 1.8.8) Version conflict: package
> > > > org.yaml:snakeyaml:jar needed in 2 different
> > > > versions: (1.10, 1.17)
> > > > ERROR: 17 package version conflicts identified
> > > >
> > > > So while it appears Guava is no longer a concern, there are other
> > > conflicting dependencies involved in the conflict here.
> > > >
> > > > I'm interested to hear your thoughts on this.
> > > >
> > > > Regards,
> > > > Mike
> > > >
> > > > On Wed, Jan 24, 2018 at 7:57 AM, Wahrmann, Helmut <
> > > helmut.wahrmann@rsa.com>
> > > > wrote:
> > > >
> > > >> Hi Mike,
> > > >>
> > > >> Thanks for the quick response.
> > > >>
> > > >> I fully agree that we need to take care about interop between the
> > > >> different components and dependencies.
> > > >> But if you look at the patch in FLUME-3021, it only puts in
> > > >> Elasticsearch dependencies.
> > > >>
> > > >> The issue with Guava, which you saw in FLUME-2921 is no longer
> > > >> there, because since mid-2016 a lot has changed in the Flume
> > > >> trunk and all those dependencies are fixed.
> > > >> It won't even be possible gto apply this patch anymore.
> > > >>
> > > >> The patch in FLUME-3021 works with the latest Elasticsearch
> > > >> versions, without introducing additional dependencies, which
> > > >> might cause problems to other projects.
> > > >> So I think it can be safely merged into trunk.
> > > >>
> > > >> best regards,
> > > >>
> > > >> Helmut
> > > >>
> > > >>
> > > >> -----Original Message-----
> > > >> From: Mike Percy [mailto:mpercy@apache.org]
> > > >> Sent: Dienstag, 23. Jänner 2018 20:39
> > > >> To: dev@flume.apache.org
> > > >> Subject: Re: Merging to trunk?
> > > >>
> > > >> Hi Helmut,
> > > >> Thank you for bringing this up on dev@ and thank you for the
> > > >> patch. I see there are other people people interested in this
> > > >> component upgrade
> > > as well.
> > > >>
> > > >> As you are probably aware, a Flume committer will need to approve
> > > >> the change before I gets merged to trunk.
> > > >>
> > > >> My primary concern w/ merging this would be compatibility of the
> > > >> dependencies. Flume suffers from a kind of a "dependency hell"
> > > >> because of a lack of classloading support (either via OSGI or Java
> modules).
> > > >> See
> > > >> https://issues.apache.org/jira/browse/FLUME-2293 for a tracking
> > ticket
> > > >> about that issue. What that means is that all of the components
> > > >> that Flume ships must have compatible dependencies with each
> > > >> other which makes changes like this more complex. Therefore I
> > > >> would like someone to verify that mvn dependency:tree does not
> > > >> show conflicts when run from the top level with the new patches.
> > > >> If memory serves, I believe Google Guava is likely to conflict.
> > > >>
> > > >> Also, I think that
> > > >> https://issues.apache.org/jira/browse/FLUME-3021
> > is
> > > >> a duplicate of https://issues.apache.org/jira/browse/FLUME-2921
> > > >> which has additional information about the various issues that we
> > > >> need to solve to do this upgrade.
> > > >>
> > > >> I'd be happy to discuss this issue some more on this thread.
> > > >>
> > > >> Mike
> > > >>
> > > >>
> > > >> On Tue, Jan 23, 2018 at 12:57 AM, Wahrmann, Helmut <
> > > >> helmut.wahrmann@rsa.com>
> > > >> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> Who decides if/when a patch or pull request gets merged to trunk?
> > > >>> Reason I am asking is for the Elasticsearch support. The current
> > > >>> code in trunk does not work with ES > 2.x.
> > > >>> Currently Elasticsearch is at 6.1.
> > > >>>
> > > >>> In FLUME-3021 we have several patches since March last year.
> > > >>> I have a patched Flume running at a customer since April last
> > > >>> year without any problems.
> > > >>>
> > > >>> So why not merging those changes into the trunk?
> > > >>> As I stated in my comment in FLUME-3021, it will not cause any
> > > >>> problems, cause the current trunk won't work with newer ES
> > > >>> versions
> > > >> anyhow.
> > > >>> I doubt that someone is out there still running ES 0.95.
> > > >>>
> > > >>> best regards,
> > > >>>
> > > >>> Helmut
> > > >>>
> > > >>
> > >
> > >
> > >
> >
> >
> > --
> > *Yonghao Zou*
> > *Shanghai China*
> >
> > *China Merchants Bank*
> >
> > *EN_BLOG <http://blog.haozi1994.com/>*
> > *CH_BLOG* <http://www.fxiaoqing.com/>
> >
>
RE: Merging to trunk?
Posted by "Wahrmann, Helmut" <he...@rsa.com>.
I think it is not a problem to distribute the modules together if we have maintainers for them.
It doesn't make sense to release Flume 1.9 in 2018 with support for module versions which were current in 2013.
Best way would be to identify people willing to upgrade the modules to their latest version.
If we can't find someone, we should distribute it separately.
best regards,
Helmut
-----Original Message-----
From: Mike Percy [mailto:mpercy@apache.org]
Sent: Montag, 29. Jänner 2018 09:12
To: dev@flume.apache.org
Subject: Re: Merging to trunk?
Interesting proposal Yonghao.
I think it's worth pointing out that today, a Flume build will work "out of the box" with any combination of supported plugins since they are all already in the Flume classpath. The downside of that, of course, is that the dependency hell makes it very hard to upgrade modules. The benefit is that it's nice from a usability perspective.
However, if we are willing to drop this "out of the box" usage feature, then we could allow for separate modules to have conflicting dependencies (i.e. Solr and ElasticSearch) as long as they are not loaded in the same Flume agent.
What do you folks think about removing the Solr Sink and the ElasticSearch Sink from the default distribution and instead distribute them as separate
(fat) jars? If we do it for those two then it might also make sense to do it for other modules as well.
I am not sure this is very user friendly in the common case where people just want to use the HDFS and HBase sink out of the box. So maybe we leave those as part of the "default" module setup?
Obviously, changing how modules get included into the classpath would be a breaking change. I think a change that drastic would call for a Flume 2.0.0 release in order to indicate that it's not backwards compatible.
Thoughts?
Mike
On Sat, Jan 27, 2018 at 10:50 PM, Yonghao Zou <yo...@gmail.com>
wrote:
> Maybe the Maven shade plugin can solve this, we can package each
> module to a fat jar and they will not depend on other modules.
>
> 2018-01-27 6:14 GMT+08:00 Wahrmann, Helmut <he...@rsa.com>:
>
> > fully agreed.
> > But we shouldn't do that do support a 4 years old component.
> > The goal should be to support the latest releases.
> > It is ridiculous to release flume 1.9 with elasticsearch 0.90.1
> > support, when 6.x is the current version.
> >
> >
> >
> > Best regards,
> >
> > Helmut Wahrmann
> >
> >
> > -------- Ursprüngliche Nachricht --------
> > Von: Ralph Goers <ra...@dslextreme.com>
> > Datum: 26.01.18 22:12 (GMT+01:00)
> > An: dev@flume.apache.org
> > Betreff: Re: Merging to trunk?
> >
> > The “right” way to deal with this from a Maven perspective is to
> > declare the version you want in the dependencyManagement section of
> > the parent
> pom
> > and to not specify any versions in child poms. Then use the Maven
> enforcer
> > plugin to make sure everything required has a declaration in the
> > dependencyManagement section.
> >
> > Ralph
> >
> > > On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut
> > > <helmut.wahrmann@rsa.com
> >
> > wrote:
> > >
> > > Hi Mike,
> > >
> > > This only shows that some other component needs to be updated as well.
> > > Which software relies on Lucene 4.3.0, which was released in 2013?
> > > It is most probably not used by anyone else anymore.
> > >
> > > This needs to be solved. It makes no sense to release Flume 1.9
> > > with
> > support for ElasticSearch 0.90.1.
> > > And it doesn't make sense either to release Flume 1.9 with support
> > > for
> a
> > component that relies on Lucene 4.3.0.
> > >
> > > We will always have dependency problems, if one component is
> > > updated to
> > a newer version and others not.
> > >
> > > To get rid of the conflict is probably to set "optional" to true
> > > in the
> > pom.xml.
> > > Then we don't distribute those Jars in the flume lib and the users
> > > need
> > to set their class path to point e.g. to Elasticsearch and then the
> correct
> > libs will be used.
> > > I doubt that a Elasticsearch user will also use some other
> > > component
> > with e.g. this outdated Lucene.
> > >
> > > best regards,
> > >
> > > Helmut
> > >
> > >
> > > -----Original Message-----
> > > From: Mike Percy [mailto:mpercy@apache.org]
> > > Sent: Donnerstag, 25. Jänner 2018 23:01
> > > To: dev@flume.apache.org
> > > Subject: Re: Merging to trunk?
> > >
> > > Hi Helmut,
> > > I wrote a small Perl script to identify dependency conflicts based
> > > on
> > output from mvn dependency:tree and posted it here:
> > https://gist.github.com/
> > > mpercy/39614d770864bdd0c386befd5e8a1840
> > >
> > > I ran that on the current trunk and it actually found some errors
> > > which
> > should be fixed:
> > >
> > > Version conflict: package com.google.guava:guava:jar needed in 2
> > different
> > > versions: (11.0.2, 18.0)
> > > Version conflict: package
> > > commons-httpclient:commons-httpclient:jar
> > needed in 2 different versions: (3.0.1, 3.1) Version conflict:
> > package commons-logging:commons-logging:jar needed in 2 different versions:
> > (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar
> needed
> > in 2 different versions: (1.5.4, 1.6) Version conflict: package
> > net.sf.jopt-simple:jopt-simple:jar needed in 2 different versions:
> > (3.2,
> > 4.7) Version conflict: package
> > org.codehaus.jackson:jackson-jaxrs:jar
> > needed in
> > > 2 different versions: (1.8.3, 1.8.8)
> > > ERROR: 6 package version conflicts identified
> > >
> > > However, after applying your patch on top of trunk and running it
> again,
> > it identified new conflicts:
> > >
> > > Version conflict: package com.google.guava:guava:jar needed in 2
> > different
> > > versions: (11.0.2, 18.0)
> > > Version conflict: package
> > > commons-httpclient:commons-httpclient:jar
> > needed in 2 different versions: (3.0.1, 3.1) Version conflict:
> > package commons-logging:commons-logging:jar needed in 2 different versions:
> > (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar
> needed
> > in 2 different versions: (1.5.4, 1.6) Version conflict: package
> > net.sf.jopt-simple:jopt-simple:jar needed in 3 different versions:
> > (3.2, 4.7, 5.0.2) Version conflict: package
> > org.apache.lucene:lucene- analyzers-common:jar
> > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict:
> package
> > org.apache.lucene:lucene-core:jar needed in 2 different versions:
> (4.3.0,
> > 7.1.0) Version conflict: package
> > org.apache.lucene:lucene-grouping:jar
> > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict:
> > package org.apache.lucene:lucene-highlighter:jar needed in 2 different versions:
> > (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-
> memory:jar
> > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict:
> > package org.apache.lucene:lucene-misc:jar needed in 2 different versions:
> (4.3.0,
> > 7.1.0) Version conflict: package
> > org.apache.lucene:lucene-queries:jar
> > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict:
> > package org.apache.lucene:lucene-queryparser:jar needed in 2 different versions:
> > (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-
> spatial:jar
> > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict:
> > package org.apache.lucene:lucene-suggest:jar needed in 2 different versions:
> > (4.3.0, 7.1.0) Version conflict: package
> > org.codehaus.jackson:jackson-
> jaxrs:jar
> > needed in
> > > 2 different versions: (1.8.3, 1.8.8) Version conflict: package
> > > org.yaml:snakeyaml:jar needed in 2 different
> > > versions: (1.10, 1.17)
> > > ERROR: 17 package version conflicts identified
> > >
> > > So while it appears Guava is no longer a concern, there are other
> > conflicting dependencies involved in the conflict here.
> > >
> > > I'm interested to hear your thoughts on this.
> > >
> > > Regards,
> > > Mike
> > >
> > > On Wed, Jan 24, 2018 at 7:57 AM, Wahrmann, Helmut <
> > helmut.wahrmann@rsa.com>
> > > wrote:
> > >
> > >> Hi Mike,
> > >>
> > >> Thanks for the quick response.
> > >>
> > >> I fully agree that we need to take care about interop between the
> > >> different components and dependencies.
> > >> But if you look at the patch in FLUME-3021, it only puts in
> > >> Elasticsearch dependencies.
> > >>
> > >> The issue with Guava, which you saw in FLUME-2921 is no longer
> > >> there, because since mid-2016 a lot has changed in the Flume
> > >> trunk and all those dependencies are fixed.
> > >> It won't even be possible gto apply this patch anymore.
> > >>
> > >> The patch in FLUME-3021 works with the latest Elasticsearch
> > >> versions, without introducing additional dependencies, which
> > >> might cause problems to other projects.
> > >> So I think it can be safely merged into trunk.
> > >>
> > >> best regards,
> > >>
> > >> Helmut
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Mike Percy [mailto:mpercy@apache.org]
> > >> Sent: Dienstag, 23. Jänner 2018 20:39
> > >> To: dev@flume.apache.org
> > >> Subject: Re: Merging to trunk?
> > >>
> > >> Hi Helmut,
> > >> Thank you for bringing this up on dev@ and thank you for the
> > >> patch. I see there are other people people interested in this
> > >> component upgrade
> > as well.
> > >>
> > >> As you are probably aware, a Flume committer will need to approve
> > >> the change before I gets merged to trunk.
> > >>
> > >> My primary concern w/ merging this would be compatibility of the
> > >> dependencies. Flume suffers from a kind of a "dependency hell"
> > >> because of a lack of classloading support (either via OSGI or Java modules).
> > >> See
> > >> https://issues.apache.org/jira/browse/FLUME-2293 for a tracking
> ticket
> > >> about that issue. What that means is that all of the components
> > >> that Flume ships must have compatible dependencies with each
> > >> other which makes changes like this more complex. Therefore I
> > >> would like someone to verify that mvn dependency:tree does not
> > >> show conflicts when run from the top level with the new patches.
> > >> If memory serves, I believe Google Guava is likely to conflict.
> > >>
> > >> Also, I think that
> > >> https://issues.apache.org/jira/browse/FLUME-3021
> is
> > >> a duplicate of https://issues.apache.org/jira/browse/FLUME-2921
> > >> which has additional information about the various issues that we
> > >> need to solve to do this upgrade.
> > >>
> > >> I'd be happy to discuss this issue some more on this thread.
> > >>
> > >> Mike
> > >>
> > >>
> > >> On Tue, Jan 23, 2018 at 12:57 AM, Wahrmann, Helmut <
> > >> helmut.wahrmann@rsa.com>
> > >> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> Who decides if/when a patch or pull request gets merged to trunk?
> > >>> Reason I am asking is for the Elasticsearch support. The current
> > >>> code in trunk does not work with ES > 2.x.
> > >>> Currently Elasticsearch is at 6.1.
> > >>>
> > >>> In FLUME-3021 we have several patches since March last year.
> > >>> I have a patched Flume running at a customer since April last
> > >>> year without any problems.
> > >>>
> > >>> So why not merging those changes into the trunk?
> > >>> As I stated in my comment in FLUME-3021, it will not cause any
> > >>> problems, cause the current trunk won't work with newer ES
> > >>> versions
> > >> anyhow.
> > >>> I doubt that someone is out there still running ES 0.95.
> > >>>
> > >>> best regards,
> > >>>
> > >>> Helmut
> > >>>
> > >>
> >
> >
> >
>
>
> --
> *Yonghao Zou*
> *Shanghai China*
>
> *China Merchants Bank*
>
> *EN_BLOG <http://blog.haozi1994.com/>*
> *CH_BLOG* <http://www.fxiaoqing.com/>
>
Re: Merging to trunk?
Posted by Mike Percy <mp...@apache.org>.
Interesting proposal Yonghao.
I think it's worth pointing out that today, a Flume build will work "out of
the box" with any combination of supported plugins since they are all
already in the Flume classpath. The downside of that, of course, is that
the dependency hell makes it very hard to upgrade modules. The benefit is
that it's nice from a usability perspective.
However, if we are willing to drop this "out of the box" usage feature,
then we could allow for separate modules to have conflicting dependencies
(i.e. Solr and ElasticSearch) as long as they are not loaded in the same
Flume agent.
What do you folks think about removing the Solr Sink and the ElasticSearch
Sink from the default distribution and instead distribute them as separate
(fat) jars? If we do it for those two then it might also make sense to do
it for other modules as well.
I am not sure this is very user friendly in the common case where people
just want to use the HDFS and HBase sink out of the box. So maybe we leave
those as part of the "default" module setup?
Obviously, changing how modules get included into the classpath would be a
breaking change. I think a change that drastic would call for a Flume 2.0.0
release in order to indicate that it's not backwards compatible.
Thoughts?
Mike
On Sat, Jan 27, 2018 at 10:50 PM, Yonghao Zou <yo...@gmail.com>
wrote:
> Maybe the Maven shade plugin can solve this, we can package each module to
> a fat jar and they will not depend on other modules.
>
> 2018-01-27 6:14 GMT+08:00 Wahrmann, Helmut <he...@rsa.com>:
>
> > fully agreed.
> > But we shouldn't do that do support a 4 years old component.
> > The goal should be to support the latest releases.
> > It is ridiculous to release flume 1.9 with elasticsearch 0.90.1 support,
> > when 6.x is the current version.
> >
> >
> >
> > Best regards,
> >
> > Helmut Wahrmann
> >
> >
> > -------- Ursprüngliche Nachricht --------
> > Von: Ralph Goers <ra...@dslextreme.com>
> > Datum: 26.01.18 22:12 (GMT+01:00)
> > An: dev@flume.apache.org
> > Betreff: Re: Merging to trunk?
> >
> > The “right” way to deal with this from a Maven perspective is to declare
> > the version you want in the dependencyManagement section of the parent
> pom
> > and to not specify any versions in child poms. Then use the Maven
> enforcer
> > plugin to make sure everything required has a declaration in the
> > dependencyManagement section.
> >
> > Ralph
> >
> > > On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut <helmut.wahrmann@rsa.com
> >
> > wrote:
> > >
> > > Hi Mike,
> > >
> > > This only shows that some other component needs to be updated as well.
> > > Which software relies on Lucene 4.3.0, which was released in 2013?
> > > It is most probably not used by anyone else anymore.
> > >
> > > This needs to be solved. It makes no sense to release Flume 1.9 with
> > support for ElasticSearch 0.90.1.
> > > And it doesn't make sense either to release Flume 1.9 with support for
> a
> > component that relies on Lucene 4.3.0.
> > >
> > > We will always have dependency problems, if one component is updated to
> > a newer version and others not.
> > >
> > > To get rid of the conflict is probably to set "optional" to true in the
> > pom.xml.
> > > Then we don't distribute those Jars in the flume lib and the users need
> > to set their class path to point e.g. to Elasticsearch and then the
> correct
> > libs will be used.
> > > I doubt that a Elasticsearch user will also use some other component
> > with e.g. this outdated Lucene.
> > >
> > > best regards,
> > >
> > > Helmut
> > >
> > >
> > > -----Original Message-----
> > > From: Mike Percy [mailto:mpercy@apache.org]
> > > Sent: Donnerstag, 25. Jänner 2018 23:01
> > > To: dev@flume.apache.org
> > > Subject: Re: Merging to trunk?
> > >
> > > Hi Helmut,
> > > I wrote a small Perl script to identify dependency conflicts based on
> > output from mvn dependency:tree and posted it here:
> > https://gist.github.com/
> > > mpercy/39614d770864bdd0c386befd5e8a1840
> > >
> > > I ran that on the current trunk and it actually found some errors which
> > should be fixed:
> > >
> > > Version conflict: package com.google.guava:guava:jar needed in 2
> > different
> > > versions: (11.0.2, 18.0)
> > > Version conflict: package commons-httpclient:commons-httpclient:jar
> > needed in 2 different versions: (3.0.1, 3.1) Version conflict: package
> > commons-logging:commons-logging:jar needed in 2 different versions:
> > (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar
> needed
> > in 2 different versions: (1.5.4, 1.6) Version conflict: package
> > net.sf.jopt-simple:jopt-simple:jar needed in 2 different versions: (3.2,
> > 4.7) Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar
> > needed in
> > > 2 different versions: (1.8.3, 1.8.8)
> > > ERROR: 6 package version conflicts identified
> > >
> > > However, after applying your patch on top of trunk and running it
> again,
> > it identified new conflicts:
> > >
> > > Version conflict: package com.google.guava:guava:jar needed in 2
> > different
> > > versions: (11.0.2, 18.0)
> > > Version conflict: package commons-httpclient:commons-httpclient:jar
> > needed in 2 different versions: (3.0.1, 3.1) Version conflict: package
> > commons-logging:commons-logging:jar needed in 2 different versions:
> > (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar
> needed
> > in 2 different versions: (1.5.4, 1.6) Version conflict: package
> > net.sf.jopt-simple:jopt-simple:jar needed in 3 different versions: (3.2,
> > 4.7, 5.0.2) Version conflict: package org.apache.lucene:lucene-
> > analyzers-common:jar
> > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict:
> package
> > org.apache.lucene:lucene-core:jar needed in 2 different versions:
> (4.3.0,
> > 7.1.0) Version conflict: package org.apache.lucene:lucene-grouping:jar
> > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package
> > org.apache.lucene:lucene-highlighter:jar needed in 2 different versions:
> > (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-
> memory:jar
> > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package
> > org.apache.lucene:lucene-misc:jar needed in 2 different versions:
> (4.3.0,
> > 7.1.0) Version conflict: package org.apache.lucene:lucene-queries:jar
> > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package
> > org.apache.lucene:lucene-queryparser:jar needed in 2 different versions:
> > (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-
> spatial:jar
> > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package
> > org.apache.lucene:lucene-suggest:jar needed in 2 different versions:
> > (4.3.0, 7.1.0) Version conflict: package org.codehaus.jackson:jackson-
> jaxrs:jar
> > needed in
> > > 2 different versions: (1.8.3, 1.8.8)
> > > Version conflict: package org.yaml:snakeyaml:jar needed in 2 different
> > > versions: (1.10, 1.17)
> > > ERROR: 17 package version conflicts identified
> > >
> > > So while it appears Guava is no longer a concern, there are other
> > conflicting dependencies involved in the conflict here.
> > >
> > > I'm interested to hear your thoughts on this.
> > >
> > > Regards,
> > > Mike
> > >
> > > On Wed, Jan 24, 2018 at 7:57 AM, Wahrmann, Helmut <
> > helmut.wahrmann@rsa.com>
> > > wrote:
> > >
> > >> Hi Mike,
> > >>
> > >> Thanks for the quick response.
> > >>
> > >> I fully agree that we need to take care about interop between the
> > >> different components and dependencies.
> > >> But if you look at the patch in FLUME-3021, it only puts in
> > >> Elasticsearch dependencies.
> > >>
> > >> The issue with Guava, which you saw in FLUME-2921 is no longer there,
> > >> because since mid-2016 a lot has changed in the Flume trunk and all
> > >> those dependencies are fixed.
> > >> It won't even be possible gto apply this patch anymore.
> > >>
> > >> The patch in FLUME-3021 works with the latest Elasticsearch versions,
> > >> without introducing additional dependencies, which might cause
> > >> problems to other projects.
> > >> So I think it can be safely merged into trunk.
> > >>
> > >> best regards,
> > >>
> > >> Helmut
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Mike Percy [mailto:mpercy@apache.org]
> > >> Sent: Dienstag, 23. Jänner 2018 20:39
> > >> To: dev@flume.apache.org
> > >> Subject: Re: Merging to trunk?
> > >>
> > >> Hi Helmut,
> > >> Thank you for bringing this up on dev@ and thank you for the patch. I
> > >> see there are other people people interested in this component upgrade
> > as well.
> > >>
> > >> As you are probably aware, a Flume committer will need to approve the
> > >> change before I gets merged to trunk.
> > >>
> > >> My primary concern w/ merging this would be compatibility of the
> > >> dependencies. Flume suffers from a kind of a "dependency hell" because
> > >> of a lack of classloading support (either via OSGI or Java modules).
> > >> See
> > >> https://issues.apache.org/jira/browse/FLUME-2293 for a tracking
> ticket
> > >> about that issue. What that means is that all of the components that
> > >> Flume ships must have compatible dependencies with each other which
> > >> makes changes like this more complex. Therefore I would like someone
> > >> to verify that mvn dependency:tree does not show conflicts when run
> > >> from the top level with the new patches. If memory serves, I believe
> > >> Google Guava is likely to conflict.
> > >>
> > >> Also, I think that https://issues.apache.org/jira/browse/FLUME-3021
> is
> > >> a duplicate of https://issues.apache.org/jira/browse/FLUME-2921 which
> > >> has additional information about the various issues that we need to
> > >> solve to do this upgrade.
> > >>
> > >> I'd be happy to discuss this issue some more on this thread.
> > >>
> > >> Mike
> > >>
> > >>
> > >> On Tue, Jan 23, 2018 at 12:57 AM, Wahrmann, Helmut <
> > >> helmut.wahrmann@rsa.com>
> > >> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> Who decides if/when a patch or pull request gets merged to trunk?
> > >>> Reason I am asking is for the Elasticsearch support. The current
> > >>> code in trunk does not work with ES > 2.x.
> > >>> Currently Elasticsearch is at 6.1.
> > >>>
> > >>> In FLUME-3021 we have several patches since March last year.
> > >>> I have a patched Flume running at a customer since April last year
> > >>> without any problems.
> > >>>
> > >>> So why not merging those changes into the trunk?
> > >>> As I stated in my comment in FLUME-3021, it will not cause any
> > >>> problems, cause the current trunk won't work with newer ES versions
> > >> anyhow.
> > >>> I doubt that someone is out there still running ES 0.95.
> > >>>
> > >>> best regards,
> > >>>
> > >>> Helmut
> > >>>
> > >>
> >
> >
> >
>
>
> --
> *Yonghao Zou*
> *Shanghai China*
>
> *China Merchants Bank*
>
> *EN_BLOG <http://blog.haozi1994.com/>*
> *CH_BLOG* <http://www.fxiaoqing.com/>
>
Re: Merging to trunk?
Posted by Yonghao Zou <yo...@gmail.com>.
Maybe the Maven shade plugin can solve this, we can package each module to
a fat jar and they will not depend on other modules.
2018-01-27 6:14 GMT+08:00 Wahrmann, Helmut <he...@rsa.com>:
> fully agreed.
> But we shouldn't do that do support a 4 years old component.
> The goal should be to support the latest releases.
> It is ridiculous to release flume 1.9 with elasticsearch 0.90.1 support,
> when 6.x is the current version.
>
>
>
> Best regards,
>
> Helmut Wahrmann
>
>
> -------- Ursprüngliche Nachricht --------
> Von: Ralph Goers <ra...@dslextreme.com>
> Datum: 26.01.18 22:12 (GMT+01:00)
> An: dev@flume.apache.org
> Betreff: Re: Merging to trunk?
>
> The “right” way to deal with this from a Maven perspective is to declare
> the version you want in the dependencyManagement section of the parent pom
> and to not specify any versions in child poms. Then use the Maven enforcer
> plugin to make sure everything required has a declaration in the
> dependencyManagement section.
>
> Ralph
>
> > On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut <he...@rsa.com>
> wrote:
> >
> > Hi Mike,
> >
> > This only shows that some other component needs to be updated as well.
> > Which software relies on Lucene 4.3.0, which was released in 2013?
> > It is most probably not used by anyone else anymore.
> >
> > This needs to be solved. It makes no sense to release Flume 1.9 with
> support for ElasticSearch 0.90.1.
> > And it doesn't make sense either to release Flume 1.9 with support for a
> component that relies on Lucene 4.3.0.
> >
> > We will always have dependency problems, if one component is updated to
> a newer version and others not.
> >
> > To get rid of the conflict is probably to set "optional" to true in the
> pom.xml.
> > Then we don't distribute those Jars in the flume lib and the users need
> to set their class path to point e.g. to Elasticsearch and then the correct
> libs will be used.
> > I doubt that a Elasticsearch user will also use some other component
> with e.g. this outdated Lucene.
> >
> > best regards,
> >
> > Helmut
> >
> >
> > -----Original Message-----
> > From: Mike Percy [mailto:mpercy@apache.org]
> > Sent: Donnerstag, 25. Jänner 2018 23:01
> > To: dev@flume.apache.org
> > Subject: Re: Merging to trunk?
> >
> > Hi Helmut,
> > I wrote a small Perl script to identify dependency conflicts based on
> output from mvn dependency:tree and posted it here:
> https://gist.github.com/
> > mpercy/39614d770864bdd0c386befd5e8a1840
> >
> > I ran that on the current trunk and it actually found some errors which
> should be fixed:
> >
> > Version conflict: package com.google.guava:guava:jar needed in 2
> different
> > versions: (11.0.2, 18.0)
> > Version conflict: package commons-httpclient:commons-httpclient:jar
> needed in 2 different versions: (3.0.1, 3.1) Version conflict: package
> commons-logging:commons-logging:jar needed in 2 different versions:
> (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar needed
> in 2 different versions: (1.5.4, 1.6) Version conflict: package
> net.sf.jopt-simple:jopt-simple:jar needed in 2 different versions: (3.2,
> 4.7) Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar
> needed in
> > 2 different versions: (1.8.3, 1.8.8)
> > ERROR: 6 package version conflicts identified
> >
> > However, after applying your patch on top of trunk and running it again,
> it identified new conflicts:
> >
> > Version conflict: package com.google.guava:guava:jar needed in 2
> different
> > versions: (11.0.2, 18.0)
> > Version conflict: package commons-httpclient:commons-httpclient:jar
> needed in 2 different versions: (3.0.1, 3.1) Version conflict: package
> commons-logging:commons-logging:jar needed in 2 different versions:
> (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar needed
> in 2 different versions: (1.5.4, 1.6) Version conflict: package
> net.sf.jopt-simple:jopt-simple:jar needed in 3 different versions: (3.2,
> 4.7, 5.0.2) Version conflict: package org.apache.lucene:lucene-
> analyzers-common:jar
> > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package
> org.apache.lucene:lucene-core:jar needed in 2 different versions: (4.3.0,
> 7.1.0) Version conflict: package org.apache.lucene:lucene-grouping:jar
> needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package
> org.apache.lucene:lucene-highlighter:jar needed in 2 different versions:
> (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-memory:jar
> needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package
> org.apache.lucene:lucene-misc:jar needed in 2 different versions: (4.3.0,
> 7.1.0) Version conflict: package org.apache.lucene:lucene-queries:jar
> needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package
> org.apache.lucene:lucene-queryparser:jar needed in 2 different versions:
> (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-spatial:jar
> needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package
> org.apache.lucene:lucene-suggest:jar needed in 2 different versions:
> (4.3.0, 7.1.0) Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar
> needed in
> > 2 different versions: (1.8.3, 1.8.8)
> > Version conflict: package org.yaml:snakeyaml:jar needed in 2 different
> > versions: (1.10, 1.17)
> > ERROR: 17 package version conflicts identified
> >
> > So while it appears Guava is no longer a concern, there are other
> conflicting dependencies involved in the conflict here.
> >
> > I'm interested to hear your thoughts on this.
> >
> > Regards,
> > Mike
> >
> > On Wed, Jan 24, 2018 at 7:57 AM, Wahrmann, Helmut <
> helmut.wahrmann@rsa.com>
> > wrote:
> >
> >> Hi Mike,
> >>
> >> Thanks for the quick response.
> >>
> >> I fully agree that we need to take care about interop between the
> >> different components and dependencies.
> >> But if you look at the patch in FLUME-3021, it only puts in
> >> Elasticsearch dependencies.
> >>
> >> The issue with Guava, which you saw in FLUME-2921 is no longer there,
> >> because since mid-2016 a lot has changed in the Flume trunk and all
> >> those dependencies are fixed.
> >> It won't even be possible gto apply this patch anymore.
> >>
> >> The patch in FLUME-3021 works with the latest Elasticsearch versions,
> >> without introducing additional dependencies, which might cause
> >> problems to other projects.
> >> So I think it can be safely merged into trunk.
> >>
> >> best regards,
> >>
> >> Helmut
> >>
> >>
> >> -----Original Message-----
> >> From: Mike Percy [mailto:mpercy@apache.org]
> >> Sent: Dienstag, 23. Jänner 2018 20:39
> >> To: dev@flume.apache.org
> >> Subject: Re: Merging to trunk?
> >>
> >> Hi Helmut,
> >> Thank you for bringing this up on dev@ and thank you for the patch. I
> >> see there are other people people interested in this component upgrade
> as well.
> >>
> >> As you are probably aware, a Flume committer will need to approve the
> >> change before I gets merged to trunk.
> >>
> >> My primary concern w/ merging this would be compatibility of the
> >> dependencies. Flume suffers from a kind of a "dependency hell" because
> >> of a lack of classloading support (either via OSGI or Java modules).
> >> See
> >> https://issues.apache.org/jira/browse/FLUME-2293 for a tracking ticket
> >> about that issue. What that means is that all of the components that
> >> Flume ships must have compatible dependencies with each other which
> >> makes changes like this more complex. Therefore I would like someone
> >> to verify that mvn dependency:tree does not show conflicts when run
> >> from the top level with the new patches. If memory serves, I believe
> >> Google Guava is likely to conflict.
> >>
> >> Also, I think that https://issues.apache.org/jira/browse/FLUME-3021 is
> >> a duplicate of https://issues.apache.org/jira/browse/FLUME-2921 which
> >> has additional information about the various issues that we need to
> >> solve to do this upgrade.
> >>
> >> I'd be happy to discuss this issue some more on this thread.
> >>
> >> Mike
> >>
> >>
> >> On Tue, Jan 23, 2018 at 12:57 AM, Wahrmann, Helmut <
> >> helmut.wahrmann@rsa.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Who decides if/when a patch or pull request gets merged to trunk?
> >>> Reason I am asking is for the Elasticsearch support. The current
> >>> code in trunk does not work with ES > 2.x.
> >>> Currently Elasticsearch is at 6.1.
> >>>
> >>> In FLUME-3021 we have several patches since March last year.
> >>> I have a patched Flume running at a customer since April last year
> >>> without any problems.
> >>>
> >>> So why not merging those changes into the trunk?
> >>> As I stated in my comment in FLUME-3021, it will not cause any
> >>> problems, cause the current trunk won't work with newer ES versions
> >> anyhow.
> >>> I doubt that someone is out there still running ES 0.95.
> >>>
> >>> best regards,
> >>>
> >>> Helmut
> >>>
> >>
>
>
>
--
*Yonghao Zou*
*Shanghai China*
*China Merchants Bank*
*EN_BLOG <http://blog.haozi1994.com/>*
*CH_BLOG* <http://www.fxiaoqing.com/>
Re: Merging to trunk?
Posted by "Wahrmann, Helmut" <he...@rsa.com>.
fully agreed.
But we shouldn't do that do support a 4 years old component.
The goal should be to support the latest releases.
It is ridiculous to release flume 1.9 with elasticsearch 0.90.1 support, when 6.x is the current version.
Best regards,
Helmut Wahrmann
-------- Ursprüngliche Nachricht --------
Von: Ralph Goers <ra...@dslextreme.com>
Datum: 26.01.18 22:12 (GMT+01:00)
An: dev@flume.apache.org
Betreff: Re: Merging to trunk?
The “right” way to deal with this from a Maven perspective is to declare the version you want in the dependencyManagement section of the parent pom and to not specify any versions in child poms. Then use the Maven enforcer plugin to make sure everything required has a declaration in the dependencyManagement section.
Ralph
> On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut <he...@rsa.com> wrote:
>
> Hi Mike,
>
> This only shows that some other component needs to be updated as well.
> Which software relies on Lucene 4.3.0, which was released in 2013?
> It is most probably not used by anyone else anymore.
>
> This needs to be solved. It makes no sense to release Flume 1.9 with support for ElasticSearch 0.90.1.
> And it doesn't make sense either to release Flume 1.9 with support for a component that relies on Lucene 4.3.0.
>
> We will always have dependency problems, if one component is updated to a newer version and others not.
>
> To get rid of the conflict is probably to set "optional" to true in the pom.xml.
> Then we don't distribute those Jars in the flume lib and the users need to set their class path to point e.g. to Elasticsearch and then the correct libs will be used.
> I doubt that a Elasticsearch user will also use some other component with e.g. this outdated Lucene.
>
> best regards,
>
> Helmut
>
>
> -----Original Message-----
> From: Mike Percy [mailto:mpercy@apache.org]
> Sent: Donnerstag, 25. Jänner 2018 23:01
> To: dev@flume.apache.org
> Subject: Re: Merging to trunk?
>
> Hi Helmut,
> I wrote a small Perl script to identify dependency conflicts based on output from mvn dependency:tree and posted it here: https://gist.github.com/
> mpercy/39614d770864bdd0c386befd5e8a1840
>
> I ran that on the current trunk and it actually found some errors which should be fixed:
>
> Version conflict: package com.google.guava:guava:jar needed in 2 different
> versions: (11.0.2, 18.0)
> Version conflict: package commons-httpclient:commons-httpclient:jar needed in 2 different versions: (3.0.1, 3.1) Version conflict: package commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 different versions: (1.5.4, 1.6) Version conflict: package net.sf.jopt-simple:jopt-simple:jar needed in 2 different versions: (3.2, 4.7) Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in
> 2 different versions: (1.8.3, 1.8.8)
> ERROR: 6 package version conflicts identified
>
> However, after applying your patch on top of trunk and running it again, it identified new conflicts:
>
> Version conflict: package com.google.guava:guava:jar needed in 2 different
> versions: (11.0.2, 18.0)
> Version conflict: package commons-httpclient:commons-httpclient:jar needed in 2 different versions: (3.0.1, 3.1) Version conflict: package commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 different versions: (1.5.4, 1.6) Version conflict: package net.sf.jopt-simple:jopt-simple:jar needed in 3 different versions: (3.2, 4.7, 5.0.2) Version conflict: package org.apache.lucene:lucene-analyzers-common:jar
> needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-core:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-grouping:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-highlighter:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-memory:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-misc:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-queries:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-queryparser:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-spatial:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-suggest:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in
> 2 different versions: (1.8.3, 1.8.8)
> Version conflict: package org.yaml:snakeyaml:jar needed in 2 different
> versions: (1.10, 1.17)
> ERROR: 17 package version conflicts identified
>
> So while it appears Guava is no longer a concern, there are other conflicting dependencies involved in the conflict here.
>
> I'm interested to hear your thoughts on this.
>
> Regards,
> Mike
>
> On Wed, Jan 24, 2018 at 7:57 AM, Wahrmann, Helmut <he...@rsa.com>
> wrote:
>
>> Hi Mike,
>>
>> Thanks for the quick response.
>>
>> I fully agree that we need to take care about interop between the
>> different components and dependencies.
>> But if you look at the patch in FLUME-3021, it only puts in
>> Elasticsearch dependencies.
>>
>> The issue with Guava, which you saw in FLUME-2921 is no longer there,
>> because since mid-2016 a lot has changed in the Flume trunk and all
>> those dependencies are fixed.
>> It won't even be possible gto apply this patch anymore.
>>
>> The patch in FLUME-3021 works with the latest Elasticsearch versions,
>> without introducing additional dependencies, which might cause
>> problems to other projects.
>> So I think it can be safely merged into trunk.
>>
>> best regards,
>>
>> Helmut
>>
>>
>> -----Original Message-----
>> From: Mike Percy [mailto:mpercy@apache.org]
>> Sent: Dienstag, 23. Jänner 2018 20:39
>> To: dev@flume.apache.org
>> Subject: Re: Merging to trunk?
>>
>> Hi Helmut,
>> Thank you for bringing this up on dev@ and thank you for the patch. I
>> see there are other people people interested in this component upgrade as well.
>>
>> As you are probably aware, a Flume committer will need to approve the
>> change before I gets merged to trunk.
>>
>> My primary concern w/ merging this would be compatibility of the
>> dependencies. Flume suffers from a kind of a "dependency hell" because
>> of a lack of classloading support (either via OSGI or Java modules).
>> See
>> https://issues.apache.org/jira/browse/FLUME-2293 for a tracking ticket
>> about that issue. What that means is that all of the components that
>> Flume ships must have compatible dependencies with each other which
>> makes changes like this more complex. Therefore I would like someone
>> to verify that mvn dependency:tree does not show conflicts when run
>> from the top level with the new patches. If memory serves, I believe
>> Google Guava is likely to conflict.
>>
>> Also, I think that https://issues.apache.org/jira/browse/FLUME-3021 is
>> a duplicate of https://issues.apache.org/jira/browse/FLUME-2921 which
>> has additional information about the various issues that we need to
>> solve to do this upgrade.
>>
>> I'd be happy to discuss this issue some more on this thread.
>>
>> Mike
>>
>>
>> On Tue, Jan 23, 2018 at 12:57 AM, Wahrmann, Helmut <
>> helmut.wahrmann@rsa.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Who decides if/when a patch or pull request gets merged to trunk?
>>> Reason I am asking is for the Elasticsearch support. The current
>>> code in trunk does not work with ES > 2.x.
>>> Currently Elasticsearch is at 6.1.
>>>
>>> In FLUME-3021 we have several patches since March last year.
>>> I have a patched Flume running at a customer since April last year
>>> without any problems.
>>>
>>> So why not merging those changes into the trunk?
>>> As I stated in my comment in FLUME-3021, it will not cause any
>>> problems, cause the current trunk won't work with newer ES versions
>> anyhow.
>>> I doubt that someone is out there still running ES 0.95.
>>>
>>> best regards,
>>>
>>> Helmut
>>>
>>
Re: Merging to trunk?
Posted by Ralph Goers <ra...@dslextreme.com>.
The “right” way to deal with this from a Maven perspective is to declare the version you want in the dependencyManagement section of the parent pom and to not specify any versions in child poms. Then use the Maven enforcer plugin to make sure everything required has a declaration in the dependencyManagement section.
Ralph
> On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut <he...@rsa.com> wrote:
>
> Hi Mike,
>
> This only shows that some other component needs to be updated as well.
> Which software relies on Lucene 4.3.0, which was released in 2013?
> It is most probably not used by anyone else anymore.
>
> This needs to be solved. It makes no sense to release Flume 1.9 with support for ElasticSearch 0.90.1.
> And it doesn't make sense either to release Flume 1.9 with support for a component that relies on Lucene 4.3.0.
>
> We will always have dependency problems, if one component is updated to a newer version and others not.
>
> To get rid of the conflict is probably to set "optional" to true in the pom.xml.
> Then we don't distribute those Jars in the flume lib and the users need to set their class path to point e.g. to Elasticsearch and then the correct libs will be used.
> I doubt that a Elasticsearch user will also use some other component with e.g. this outdated Lucene.
>
> best regards,
>
> Helmut
>
>
> -----Original Message-----
> From: Mike Percy [mailto:mpercy@apache.org]
> Sent: Donnerstag, 25. Jänner 2018 23:01
> To: dev@flume.apache.org
> Subject: Re: Merging to trunk?
>
> Hi Helmut,
> I wrote a small Perl script to identify dependency conflicts based on output from mvn dependency:tree and posted it here: https://gist.github.com/
> mpercy/39614d770864bdd0c386befd5e8a1840
>
> I ran that on the current trunk and it actually found some errors which should be fixed:
>
> Version conflict: package com.google.guava:guava:jar needed in 2 different
> versions: (11.0.2, 18.0)
> Version conflict: package commons-httpclient:commons-httpclient:jar needed in 2 different versions: (3.0.1, 3.1) Version conflict: package commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 different versions: (1.5.4, 1.6) Version conflict: package net.sf.jopt-simple:jopt-simple:jar needed in 2 different versions: (3.2, 4.7) Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in
> 2 different versions: (1.8.3, 1.8.8)
> ERROR: 6 package version conflicts identified
>
> However, after applying your patch on top of trunk and running it again, it identified new conflicts:
>
> Version conflict: package com.google.guava:guava:jar needed in 2 different
> versions: (11.0.2, 18.0)
> Version conflict: package commons-httpclient:commons-httpclient:jar needed in 2 different versions: (3.0.1, 3.1) Version conflict: package commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 different versions: (1.5.4, 1.6) Version conflict: package net.sf.jopt-simple:jopt-simple:jar needed in 3 different versions: (3.2, 4.7, 5.0.2) Version conflict: package org.apache.lucene:lucene-analyzers-common:jar
> needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-core:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-grouping:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-highlighter:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-memory:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-misc:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-queries:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-queryparser:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-spatial:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-suggest:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in
> 2 different versions: (1.8.3, 1.8.8)
> Version conflict: package org.yaml:snakeyaml:jar needed in 2 different
> versions: (1.10, 1.17)
> ERROR: 17 package version conflicts identified
>
> So while it appears Guava is no longer a concern, there are other conflicting dependencies involved in the conflict here.
>
> I'm interested to hear your thoughts on this.
>
> Regards,
> Mike
>
> On Wed, Jan 24, 2018 at 7:57 AM, Wahrmann, Helmut <he...@rsa.com>
> wrote:
>
>> Hi Mike,
>>
>> Thanks for the quick response.
>>
>> I fully agree that we need to take care about interop between the
>> different components and dependencies.
>> But if you look at the patch in FLUME-3021, it only puts in
>> Elasticsearch dependencies.
>>
>> The issue with Guava, which you saw in FLUME-2921 is no longer there,
>> because since mid-2016 a lot has changed in the Flume trunk and all
>> those dependencies are fixed.
>> It won't even be possible gto apply this patch anymore.
>>
>> The patch in FLUME-3021 works with the latest Elasticsearch versions,
>> without introducing additional dependencies, which might cause
>> problems to other projects.
>> So I think it can be safely merged into trunk.
>>
>> best regards,
>>
>> Helmut
>>
>>
>> -----Original Message-----
>> From: Mike Percy [mailto:mpercy@apache.org]
>> Sent: Dienstag, 23. Jänner 2018 20:39
>> To: dev@flume.apache.org
>> Subject: Re: Merging to trunk?
>>
>> Hi Helmut,
>> Thank you for bringing this up on dev@ and thank you for the patch. I
>> see there are other people people interested in this component upgrade as well.
>>
>> As you are probably aware, a Flume committer will need to approve the
>> change before I gets merged to trunk.
>>
>> My primary concern w/ merging this would be compatibility of the
>> dependencies. Flume suffers from a kind of a "dependency hell" because
>> of a lack of classloading support (either via OSGI or Java modules).
>> See
>> https://issues.apache.org/jira/browse/FLUME-2293 for a tracking ticket
>> about that issue. What that means is that all of the components that
>> Flume ships must have compatible dependencies with each other which
>> makes changes like this more complex. Therefore I would like someone
>> to verify that mvn dependency:tree does not show conflicts when run
>> from the top level with the new patches. If memory serves, I believe
>> Google Guava is likely to conflict.
>>
>> Also, I think that https://issues.apache.org/jira/browse/FLUME-3021 is
>> a duplicate of https://issues.apache.org/jira/browse/FLUME-2921 which
>> has additional information about the various issues that we need to
>> solve to do this upgrade.
>>
>> I'd be happy to discuss this issue some more on this thread.
>>
>> Mike
>>
>>
>> On Tue, Jan 23, 2018 at 12:57 AM, Wahrmann, Helmut <
>> helmut.wahrmann@rsa.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Who decides if/when a patch or pull request gets merged to trunk?
>>> Reason I am asking is for the Elasticsearch support. The current
>>> code in trunk does not work with ES > 2.x.
>>> Currently Elasticsearch is at 6.1.
>>>
>>> In FLUME-3021 we have several patches since March last year.
>>> I have a patched Flume running at a customer since April last year
>>> without any problems.
>>>
>>> So why not merging those changes into the trunk?
>>> As I stated in my comment in FLUME-3021, it will not cause any
>>> problems, cause the current trunk won't work with newer ES versions
>> anyhow.
>>> I doubt that someone is out there still running ES 0.95.
>>>
>>> best regards,
>>>
>>> Helmut
>>>
>>
RE: Merging to trunk?
Posted by "Wahrmann, Helmut" <he...@rsa.com>.
Hi Mike,
This only shows that some other component needs to be updated as well.
Which software relies on Lucene 4.3.0, which was released in 2013?
It is most probably not used by anyone else anymore.
This needs to be solved. It makes no sense to release Flume 1.9 with support for ElasticSearch 0.90.1.
And it doesn't make sense either to release Flume 1.9 with support for a component that relies on Lucene 4.3.0.
We will always have dependency problems, if one component is updated to a newer version and others not.
To get rid of the conflict is probably to set "optional" to true in the pom.xml.
Then we don't distribute those Jars in the flume lib and the users need to set their class path to point e.g. to Elasticsearch and then the correct libs will be used.
I doubt that a Elasticsearch user will also use some other component with e.g. this outdated Lucene.
best regards,
Helmut
-----Original Message-----
From: Mike Percy [mailto:mpercy@apache.org]
Sent: Donnerstag, 25. Jänner 2018 23:01
To: dev@flume.apache.org
Subject: Re: Merging to trunk?
Hi Helmut,
I wrote a small Perl script to identify dependency conflicts based on output from mvn dependency:tree and posted it here: https://gist.github.com/
mpercy/39614d770864bdd0c386befd5e8a1840
I ran that on the current trunk and it actually found some errors which should be fixed:
Version conflict: package com.google.guava:guava:jar needed in 2 different
versions: (11.0.2, 18.0)
Version conflict: package commons-httpclient:commons-httpclient:jar needed in 2 different versions: (3.0.1, 3.1) Version conflict: package commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 different versions: (1.5.4, 1.6) Version conflict: package net.sf.jopt-simple:jopt-simple:jar needed in 2 different versions: (3.2, 4.7) Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in
2 different versions: (1.8.3, 1.8.8)
ERROR: 6 package version conflicts identified
However, after applying your patch on top of trunk and running it again, it identified new conflicts:
Version conflict: package com.google.guava:guava:jar needed in 2 different
versions: (11.0.2, 18.0)
Version conflict: package commons-httpclient:commons-httpclient:jar needed in 2 different versions: (3.0.1, 3.1) Version conflict: package commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 different versions: (1.5.4, 1.6) Version conflict: package net.sf.jopt-simple:jopt-simple:jar needed in 3 different versions: (3.2, 4.7, 5.0.2) Version conflict: package org.apache.lucene:lucene-analyzers-common:jar
needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-core:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-grouping:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-highlighter:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-memory:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-misc:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-queries:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-queryparser:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-spatial:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-suggest:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in
2 different versions: (1.8.3, 1.8.8)
Version conflict: package org.yaml:snakeyaml:jar needed in 2 different
versions: (1.10, 1.17)
ERROR: 17 package version conflicts identified
So while it appears Guava is no longer a concern, there are other conflicting dependencies involved in the conflict here.
I'm interested to hear your thoughts on this.
Regards,
Mike
On Wed, Jan 24, 2018 at 7:57 AM, Wahrmann, Helmut <he...@rsa.com>
wrote:
> Hi Mike,
>
> Thanks for the quick response.
>
> I fully agree that we need to take care about interop between the
> different components and dependencies.
> But if you look at the patch in FLUME-3021, it only puts in
> Elasticsearch dependencies.
>
> The issue with Guava, which you saw in FLUME-2921 is no longer there,
> because since mid-2016 a lot has changed in the Flume trunk and all
> those dependencies are fixed.
> It won't even be possible gto apply this patch anymore.
>
> The patch in FLUME-3021 works with the latest Elasticsearch versions,
> without introducing additional dependencies, which might cause
> problems to other projects.
> So I think it can be safely merged into trunk.
>
> best regards,
>
> Helmut
>
>
> -----Original Message-----
> From: Mike Percy [mailto:mpercy@apache.org]
> Sent: Dienstag, 23. Jänner 2018 20:39
> To: dev@flume.apache.org
> Subject: Re: Merging to trunk?
>
> Hi Helmut,
> Thank you for bringing this up on dev@ and thank you for the patch. I
> see there are other people people interested in this component upgrade as well.
>
> As you are probably aware, a Flume committer will need to approve the
> change before I gets merged to trunk.
>
> My primary concern w/ merging this would be compatibility of the
> dependencies. Flume suffers from a kind of a "dependency hell" because
> of a lack of classloading support (either via OSGI or Java modules).
> See
> https://issues.apache.org/jira/browse/FLUME-2293 for a tracking ticket
> about that issue. What that means is that all of the components that
> Flume ships must have compatible dependencies with each other which
> makes changes like this more complex. Therefore I would like someone
> to verify that mvn dependency:tree does not show conflicts when run
> from the top level with the new patches. If memory serves, I believe
> Google Guava is likely to conflict.
>
> Also, I think that https://issues.apache.org/jira/browse/FLUME-3021 is
> a duplicate of https://issues.apache.org/jira/browse/FLUME-2921 which
> has additional information about the various issues that we need to
> solve to do this upgrade.
>
> I'd be happy to discuss this issue some more on this thread.
>
> Mike
>
>
> On Tue, Jan 23, 2018 at 12:57 AM, Wahrmann, Helmut <
> helmut.wahrmann@rsa.com>
> wrote:
>
> > Hi,
> >
> > Who decides if/when a patch or pull request gets merged to trunk?
> > Reason I am asking is for the Elasticsearch support. The current
> > code in trunk does not work with ES > 2.x.
> > Currently Elasticsearch is at 6.1.
> >
> > In FLUME-3021 we have several patches since March last year.
> > I have a patched Flume running at a customer since April last year
> > without any problems.
> >
> > So why not merging those changes into the trunk?
> > As I stated in my comment in FLUME-3021, it will not cause any
> > problems, cause the current trunk won't work with newer ES versions
> anyhow.
> > I doubt that someone is out there still running ES 0.95.
> >
> > best regards,
> >
> > Helmut
> >
>
Re: Merging to trunk?
Posted by Mike Percy <mp...@apache.org>.
Hi Helmut,
I wrote a small Perl script to identify dependency conflicts based on
output from mvn dependency:tree and posted it here: https://gist.github.com/
mpercy/39614d770864bdd0c386befd5e8a1840
I ran that on the current trunk and it actually found some errors which
should be fixed:
Version conflict: package com.google.guava:guava:jar needed in 2 different
versions: (11.0.2, 18.0)
Version conflict: package commons-httpclient:commons-httpclient:jar needed
in 2 different versions: (3.0.1, 3.1)
Version conflict: package commons-logging:commons-logging:jar needed in 2
different versions: (1.1.3, 1.2)
Version conflict: package commons-pool:commons-pool:jar needed in 2
different versions: (1.5.4, 1.6)
Version conflict: package net.sf.jopt-simple:jopt-simple:jar needed in 2
different versions: (3.2, 4.7)
Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in
2 different versions: (1.8.3, 1.8.8)
ERROR: 6 package version conflicts identified
However, after applying your patch on top of trunk and running it again, it
identified new conflicts:
Version conflict: package com.google.guava:guava:jar needed in 2 different
versions: (11.0.2, 18.0)
Version conflict: package commons-httpclient:commons-httpclient:jar needed
in 2 different versions: (3.0.1, 3.1)
Version conflict: package commons-logging:commons-logging:jar needed in 2
different versions: (1.1.3, 1.2)
Version conflict: package commons-pool:commons-pool:jar needed in 2
different versions: (1.5.4, 1.6)
Version conflict: package net.sf.jopt-simple:jopt-simple:jar needed in 3
different versions: (3.2, 4.7, 5.0.2)
Version conflict: package org.apache.lucene:lucene-analyzers-common:jar
needed in 2 different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-core:jar needed in 2
different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-grouping:jar needed in 2
different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-highlighter:jar needed
in 2 different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-memory:jar needed in 2
different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-misc:jar needed in 2
different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-queries:jar needed in 2
different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-queryparser:jar needed
in 2 different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-spatial:jar needed in 2
different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-suggest:jar needed in 2
different versions: (4.3.0, 7.1.0)
Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in
2 different versions: (1.8.3, 1.8.8)
Version conflict: package org.yaml:snakeyaml:jar needed in 2 different
versions: (1.10, 1.17)
ERROR: 17 package version conflicts identified
So while it appears Guava is no longer a concern, there are other
conflicting dependencies involved in the conflict here.
I'm interested to hear your thoughts on this.
Regards,
Mike
On Wed, Jan 24, 2018 at 7:57 AM, Wahrmann, Helmut <he...@rsa.com>
wrote:
> Hi Mike,
>
> Thanks for the quick response.
>
> I fully agree that we need to take care about interop between the
> different components and dependencies.
> But if you look at the patch in FLUME-3021, it only puts in Elasticsearch
> dependencies.
>
> The issue with Guava, which you saw in FLUME-2921 is no longer there,
> because since mid-2016 a lot has changed in the Flume trunk and all those
> dependencies are fixed.
> It won't even be possible gto apply this patch anymore.
>
> The patch in FLUME-3021 works with the latest Elasticsearch versions,
> without introducing additional dependencies, which might cause problems to
> other projects.
> So I think it can be safely merged into trunk.
>
> best regards,
>
> Helmut
>
>
> -----Original Message-----
> From: Mike Percy [mailto:mpercy@apache.org]
> Sent: Dienstag, 23. Jänner 2018 20:39
> To: dev@flume.apache.org
> Subject: Re: Merging to trunk?
>
> Hi Helmut,
> Thank you for bringing this up on dev@ and thank you for the patch. I see
> there are other people people interested in this component upgrade as well.
>
> As you are probably aware, a Flume committer will need to approve the
> change before I gets merged to trunk.
>
> My primary concern w/ merging this would be compatibility of the
> dependencies. Flume suffers from a kind of a "dependency hell" because of a
> lack of classloading support (either via OSGI or Java modules). See
> https://issues.apache.org/jira/browse/FLUME-2293 for a tracking ticket
> about that issue. What that means is that all of the components that Flume
> ships must have compatible dependencies with each other which makes changes
> like this more complex. Therefore I would like someone to verify that mvn
> dependency:tree does not show conflicts when run from the top level with
> the new patches. If memory serves, I believe Google Guava is likely to
> conflict.
>
> Also, I think that https://issues.apache.org/jira/browse/FLUME-3021 is a
> duplicate of https://issues.apache.org/jira/browse/FLUME-2921 which has
> additional information about the various issues that we need to solve to do
> this upgrade.
>
> I'd be happy to discuss this issue some more on this thread.
>
> Mike
>
>
> On Tue, Jan 23, 2018 at 12:57 AM, Wahrmann, Helmut <
> helmut.wahrmann@rsa.com>
> wrote:
>
> > Hi,
> >
> > Who decides if/when a patch or pull request gets merged to trunk?
> > Reason I am asking is for the Elasticsearch support. The current code
> > in trunk does not work with ES > 2.x.
> > Currently Elasticsearch is at 6.1.
> >
> > In FLUME-3021 we have several patches since March last year.
> > I have a patched Flume running at a customer since April last year
> > without any problems.
> >
> > So why not merging those changes into the trunk?
> > As I stated in my comment in FLUME-3021, it will not cause any
> > problems, cause the current trunk won't work with newer ES versions
> anyhow.
> > I doubt that someone is out there still running ES 0.95.
> >
> > best regards,
> >
> > Helmut
> >
>
RE: Merging to trunk?
Posted by "Wahrmann, Helmut" <he...@rsa.com>.
Hi Mike,
Thanks for the quick response.
I fully agree that we need to take care about interop between the different components and dependencies.
But if you look at the patch in FLUME-3021, it only puts in Elasticsearch dependencies.
The issue with Guava, which you saw in FLUME-2921 is no longer there, because since mid-2016 a lot has changed in the Flume trunk and all those dependencies are fixed.
It won't even be possible gto apply this patch anymore.
The patch in FLUME-3021 works with the latest Elasticsearch versions, without introducing additional dependencies, which might cause problems to other projects.
So I think it can be safely merged into trunk.
best regards,
Helmut
-----Original Message-----
From: Mike Percy [mailto:mpercy@apache.org]
Sent: Dienstag, 23. Jänner 2018 20:39
To: dev@flume.apache.org
Subject: Re: Merging to trunk?
Hi Helmut,
Thank you for bringing this up on dev@ and thank you for the patch. I see there are other people people interested in this component upgrade as well.
As you are probably aware, a Flume committer will need to approve the change before I gets merged to trunk.
My primary concern w/ merging this would be compatibility of the dependencies. Flume suffers from a kind of a "dependency hell" because of a lack of classloading support (either via OSGI or Java modules). See
https://issues.apache.org/jira/browse/FLUME-2293 for a tracking ticket about that issue. What that means is that all of the components that Flume ships must have compatible dependencies with each other which makes changes like this more complex. Therefore I would like someone to verify that mvn dependency:tree does not show conflicts when run from the top level with the new patches. If memory serves, I believe Google Guava is likely to conflict.
Also, I think that https://issues.apache.org/jira/browse/FLUME-3021 is a duplicate of https://issues.apache.org/jira/browse/FLUME-2921 which has additional information about the various issues that we need to solve to do this upgrade.
I'd be happy to discuss this issue some more on this thread.
Mike
On Tue, Jan 23, 2018 at 12:57 AM, Wahrmann, Helmut <he...@rsa.com>
wrote:
> Hi,
>
> Who decides if/when a patch or pull request gets merged to trunk?
> Reason I am asking is for the Elasticsearch support. The current code
> in trunk does not work with ES > 2.x.
> Currently Elasticsearch is at 6.1.
>
> In FLUME-3021 we have several patches since March last year.
> I have a patched Flume running at a customer since April last year
> without any problems.
>
> So why not merging those changes into the trunk?
> As I stated in my comment in FLUME-3021, it will not cause any
> problems, cause the current trunk won't work with newer ES versions anyhow.
> I doubt that someone is out there still running ES 0.95.
>
> best regards,
>
> Helmut
>
Re: Merging to trunk?
Posted by Mike Percy <mp...@apache.org>.
Hi Helmut,
Thank you for bringing this up on dev@ and thank you for the patch. I see
there are other people people interested in this component upgrade as well.
As you are probably aware, a Flume committer will need to approve the
change before I gets merged to trunk.
My primary concern w/ merging this would be compatibility of the
dependencies. Flume suffers from a kind of a "dependency hell" because of a
lack of classloading support (either via OSGI or Java modules). See
https://issues.apache.org/jira/browse/FLUME-2293 for a tracking ticket
about that issue. What that means is that all of the components that Flume
ships must have compatible dependencies with each other which makes changes
like this more complex. Therefore I would like someone to verify that mvn
dependency:tree does not show conflicts when run from the top level with
the new patches. If memory serves, I believe Google Guava is likely to
conflict.
Also, I think that https://issues.apache.org/jira/browse/FLUME-3021 is a
duplicate of https://issues.apache.org/jira/browse/FLUME-2921 which has
additional information about the various issues that we need to solve to do
this upgrade.
I'd be happy to discuss this issue some more on this thread.
Mike
On Tue, Jan 23, 2018 at 12:57 AM, Wahrmann, Helmut <he...@rsa.com>
wrote:
> Hi,
>
> Who decides if/when a patch or pull request gets merged to trunk?
> Reason I am asking is for the Elasticsearch support. The current code in
> trunk does not work with ES > 2.x.
> Currently Elasticsearch is at 6.1.
>
> In FLUME-3021 we have several patches since March last year.
> I have a patched Flume running at a customer since April last year without
> any problems.
>
> So why not merging those changes into the trunk?
> As I stated in my comment in FLUME-3021, it will not cause any problems,
> cause the current trunk won't work with newer ES versions anyhow.
> I doubt that someone is out there still running ES 0.95.
>
> best regards,
>
> Helmut
>