You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Pramod Immaneni <pr...@datatorrent.com> on 2016/07/08 18:28:01 UTC
Issue with httpclient
Hi,
We have seen an issue with different versions of
org.apache.httpcomponents:httpclient jar getting included in classpath when
apps are launched on newer versions of Hadoop. This is because we have an
explicit dependency in Apex to version 4.3.5, the version of hadoop we
build against (2.2 for backward compatibility) does not have a dependency
to this artifact but newer versions of hadoop have this dependency. To get
around this problem we are thinking of shading this dependency internally
in apex. Any other ideas or suggestions?
Thanks
Re: Issue with httpclient
Posted by Pramod Immaneni <pr...@datatorrent.com>.
We could go with a separate artifact although I think we should have a
stronger reason than builds to create one, such as the likelihood of others
needing it. If it were shaded internally, in every build the mapping would
still remain the same isn't it, so there would be no changes to things like
exception traces between versions and additionally, the incremental build
cost of this approach wouldn't be significant compared to the total build.
I am fine with going with a separate artifact. There wouldn't be much of a
loss in terms of dependency information for this one, as it has only two
dependencies one of which is part of the same project and another a logging
framework. Also, the issue could be considered a blocking issue as it
prevents launching of applications, so in the interest of expediency, I
think we should defer the idea of combining all the shaded artifacts (such
as this one and ning) into a single artifact to a later point of time.
David, Tim your thoughts on proceeding with a separate artifact.
Thanks
On Sat, Jul 9, 2016 at 12:02 PM, Thomas Weise <th...@datatorrent.com>
wrote:
> Keeping shaded dependencies as separate artifacts simplifies the build and
> allows others to use them as well. Why shade a dependency on every build
> invocation when it never changes?
>
> We could also consider building a single shaded artifact for multiple
> dependencies in case we are adding more in the future. The downside is that
> the dependency in turn may require other transitive dependencies to be
> shaded (see ning) and information is lost for dependency:tree and
> elsewhere.
>
> Thomas
>
>
> On Sat, Jul 9, 2016 at 5:32 AM, David Yan <da...@datatorrent.com> wrote:
>
> > I'm good with shading it internally in apex-engine for now until there is
> > an actual need in a real use case. Thanks!
> >
> > David
> >
> > On Fri, Jul 8, 2016 at 4:42 PM, Pramod Immaneni <pr...@datatorrent.com>
> > wrote:
> >
> > > Good question. Even though it would be useful for somebody wanting to
> use
> > > SPNEGO in their operator for example I don't see any immediate use. We
> > can
> > > just go with it being internal and create a separate artifact in future
> > if
> > > need arises. What do you think?
> > >
> > > Thanks
> > >
> > > On Fri, Jul 8, 2016 at 11:40 AM, David Yan <da...@datatorrent.com>
> > wrote:
> > >
> > > > +1 for the change. This would fix dependency conflicts between
> > httpclient
> > > > 4.2.5 and 4.3.5.
> > > > I think the question is whether we should publish a separate shaded
> > > > artifact (like we did for ning-ahc) for the shaded httpclient 4.3.5,
> or
> > > > just shading it internally in apex-engine. The former allows
> > application
> > > > and operator developers to use the shaded artifact and the latter
> hides
> > > it
> > > > completely.
> > > >
> > > > Davod
> > > >
> > > > On Fri, Jul 8, 2016 at 11:31 AM, Pramod Immaneni <
> > pramod@datatorrent.com
> > > >
> > > > wrote:
> > > >
> > > > > Forgot to add that 4.3.5 is a later version that what newer
> versions
> > of
> > > > > hadoop are using (4.2.5) and we need it for kerberos SPNEGO
> security
> > > > > related functionality.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Fri, Jul 8, 2016 at 11:28 AM, Pramod Immaneni <
> > > pramod@datatorrent.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We have seen an issue with different versions of
> > > > > > org.apache.httpcomponents:httpclient jar getting included in
> > > classpath
> > > > > when
> > > > > > apps are launched on newer versions of Hadoop. This is because we
> > > have
> > > > an
> > > > > > explicit dependency in Apex to version 4.3.5, the version of
> hadoop
> > > we
> > > > > > build against (2.2 for backward compatibility) does not have a
> > > > dependency
> > > > > > to this artifact but newer versions of hadoop have this
> dependency.
> > > To
> > > > > get
> > > > > > around this problem we are thinking of shading this dependency
> > > > internally
> > > > > > in apex. Any other ideas or suggestions?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Issue with httpclient
Posted by Thomas Weise <th...@datatorrent.com>.
Keeping shaded dependencies as separate artifacts simplifies the build and
allows others to use them as well. Why shade a dependency on every build
invocation when it never changes?
We could also consider building a single shaded artifact for multiple
dependencies in case we are adding more in the future. The downside is that
the dependency in turn may require other transitive dependencies to be
shaded (see ning) and information is lost for dependency:tree and
elsewhere.
Thomas
On Sat, Jul 9, 2016 at 5:32 AM, David Yan <da...@datatorrent.com> wrote:
> I'm good with shading it internally in apex-engine for now until there is
> an actual need in a real use case. Thanks!
>
> David
>
> On Fri, Jul 8, 2016 at 4:42 PM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > Good question. Even though it would be useful for somebody wanting to use
> > SPNEGO in their operator for example I don't see any immediate use. We
> can
> > just go with it being internal and create a separate artifact in future
> if
> > need arises. What do you think?
> >
> > Thanks
> >
> > On Fri, Jul 8, 2016 at 11:40 AM, David Yan <da...@datatorrent.com>
> wrote:
> >
> > > +1 for the change. This would fix dependency conflicts between
> httpclient
> > > 4.2.5 and 4.3.5.
> > > I think the question is whether we should publish a separate shaded
> > > artifact (like we did for ning-ahc) for the shaded httpclient 4.3.5, or
> > > just shading it internally in apex-engine. The former allows
> application
> > > and operator developers to use the shaded artifact and the latter hides
> > it
> > > completely.
> > >
> > > Davod
> > >
> > > On Fri, Jul 8, 2016 at 11:31 AM, Pramod Immaneni <
> pramod@datatorrent.com
> > >
> > > wrote:
> > >
> > > > Forgot to add that 4.3.5 is a later version that what newer versions
> of
> > > > hadoop are using (4.2.5) and we need it for kerberos SPNEGO security
> > > > related functionality.
> > > >
> > > > Thanks
> > > >
> > > > On Fri, Jul 8, 2016 at 11:28 AM, Pramod Immaneni <
> > pramod@datatorrent.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We have seen an issue with different versions of
> > > > > org.apache.httpcomponents:httpclient jar getting included in
> > classpath
> > > > when
> > > > > apps are launched on newer versions of Hadoop. This is because we
> > have
> > > an
> > > > > explicit dependency in Apex to version 4.3.5, the version of hadoop
> > we
> > > > > build against (2.2 for backward compatibility) does not have a
> > > dependency
> > > > > to this artifact but newer versions of hadoop have this dependency.
> > To
> > > > get
> > > > > around this problem we are thinking of shading this dependency
> > > internally
> > > > > in apex. Any other ideas or suggestions?
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > >
> >
>
Re: Issue with httpclient
Posted by David Yan <da...@datatorrent.com>.
I'm good with shading it internally in apex-engine for now until there is
an actual need in a real use case. Thanks!
David
On Fri, Jul 8, 2016 at 4:42 PM, Pramod Immaneni <pr...@datatorrent.com>
wrote:
> Good question. Even though it would be useful for somebody wanting to use
> SPNEGO in their operator for example I don't see any immediate use. We can
> just go with it being internal and create a separate artifact in future if
> need arises. What do you think?
>
> Thanks
>
> On Fri, Jul 8, 2016 at 11:40 AM, David Yan <da...@datatorrent.com> wrote:
>
> > +1 for the change. This would fix dependency conflicts between httpclient
> > 4.2.5 and 4.3.5.
> > I think the question is whether we should publish a separate shaded
> > artifact (like we did for ning-ahc) for the shaded httpclient 4.3.5, or
> > just shading it internally in apex-engine. The former allows application
> > and operator developers to use the shaded artifact and the latter hides
> it
> > completely.
> >
> > Davod
> >
> > On Fri, Jul 8, 2016 at 11:31 AM, Pramod Immaneni <pramod@datatorrent.com
> >
> > wrote:
> >
> > > Forgot to add that 4.3.5 is a later version that what newer versions of
> > > hadoop are using (4.2.5) and we need it for kerberos SPNEGO security
> > > related functionality.
> > >
> > > Thanks
> > >
> > > On Fri, Jul 8, 2016 at 11:28 AM, Pramod Immaneni <
> pramod@datatorrent.com
> > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > We have seen an issue with different versions of
> > > > org.apache.httpcomponents:httpclient jar getting included in
> classpath
> > > when
> > > > apps are launched on newer versions of Hadoop. This is because we
> have
> > an
> > > > explicit dependency in Apex to version 4.3.5, the version of hadoop
> we
> > > > build against (2.2 for backward compatibility) does not have a
> > dependency
> > > > to this artifact but newer versions of hadoop have this dependency.
> To
> > > get
> > > > around this problem we are thinking of shading this dependency
> > internally
> > > > in apex. Any other ideas or suggestions?
> > > >
> > > > Thanks
> > > >
> > >
> >
>
Re: Issue with httpclient
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Good question. Even though it would be useful for somebody wanting to use
SPNEGO in their operator for example I don't see any immediate use. We can
just go with it being internal and create a separate artifact in future if
need arises. What do you think?
Thanks
On Fri, Jul 8, 2016 at 11:40 AM, David Yan <da...@datatorrent.com> wrote:
> +1 for the change. This would fix dependency conflicts between httpclient
> 4.2.5 and 4.3.5.
> I think the question is whether we should publish a separate shaded
> artifact (like we did for ning-ahc) for the shaded httpclient 4.3.5, or
> just shading it internally in apex-engine. The former allows application
> and operator developers to use the shaded artifact and the latter hides it
> completely.
>
> Davod
>
> On Fri, Jul 8, 2016 at 11:31 AM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > Forgot to add that 4.3.5 is a later version that what newer versions of
> > hadoop are using (4.2.5) and we need it for kerberos SPNEGO security
> > related functionality.
> >
> > Thanks
> >
> > On Fri, Jul 8, 2016 at 11:28 AM, Pramod Immaneni <pramod@datatorrent.com
> >
> > wrote:
> >
> > > Hi,
> > >
> > > We have seen an issue with different versions of
> > > org.apache.httpcomponents:httpclient jar getting included in classpath
> > when
> > > apps are launched on newer versions of Hadoop. This is because we have
> an
> > > explicit dependency in Apex to version 4.3.5, the version of hadoop we
> > > build against (2.2 for backward compatibility) does not have a
> dependency
> > > to this artifact but newer versions of hadoop have this dependency. To
> > get
> > > around this problem we are thinking of shading this dependency
> internally
> > > in apex. Any other ideas or suggestions?
> > >
> > > Thanks
> > >
> >
>
Re: Issue with httpclient
Posted by David Yan <da...@datatorrent.com>.
+1 for the change. This would fix dependency conflicts between httpclient
4.2.5 and 4.3.5.
I think the question is whether we should publish a separate shaded
artifact (like we did for ning-ahc) for the shaded httpclient 4.3.5, or
just shading it internally in apex-engine. The former allows application
and operator developers to use the shaded artifact and the latter hides it
completely.
Davod
On Fri, Jul 8, 2016 at 11:31 AM, Pramod Immaneni <pr...@datatorrent.com>
wrote:
> Forgot to add that 4.3.5 is a later version that what newer versions of
> hadoop are using (4.2.5) and we need it for kerberos SPNEGO security
> related functionality.
>
> Thanks
>
> On Fri, Jul 8, 2016 at 11:28 AM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > Hi,
> >
> > We have seen an issue with different versions of
> > org.apache.httpcomponents:httpclient jar getting included in classpath
> when
> > apps are launched on newer versions of Hadoop. This is because we have an
> > explicit dependency in Apex to version 4.3.5, the version of hadoop we
> > build against (2.2 for backward compatibility) does not have a dependency
> > to this artifact but newer versions of hadoop have this dependency. To
> get
> > around this problem we are thinking of shading this dependency internally
> > in apex. Any other ideas or suggestions?
> >
> > Thanks
> >
>
Re: Issue with httpclient
Posted by Timothy Farkas <ti...@gmail.com>.
+1 Flink and Spark shade a lot of their dependencies to avoid problems like
this, they both even shade guava. Don't see any down side to doing it.
On Fri, Jul 8, 2016 at 11:31 AM, Pramod Immaneni <pr...@datatorrent.com>
wrote:
> Forgot to add that 4.3.5 is a later version that what newer versions of
> hadoop are using (4.2.5) and we need it for kerberos SPNEGO security
> related functionality.
>
> Thanks
>
> On Fri, Jul 8, 2016 at 11:28 AM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > Hi,
> >
> > We have seen an issue with different versions of
> > org.apache.httpcomponents:httpclient jar getting included in classpath
> when
> > apps are launched on newer versions of Hadoop. This is because we have an
> > explicit dependency in Apex to version 4.3.5, the version of hadoop we
> > build against (2.2 for backward compatibility) does not have a dependency
> > to this artifact but newer versions of hadoop have this dependency. To
> get
> > around this problem we are thinking of shading this dependency internally
> > in apex. Any other ideas or suggestions?
> >
> > Thanks
> >
>
Re: Issue with httpclient
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Forgot to add that 4.3.5 is a later version that what newer versions of
hadoop are using (4.2.5) and we need it for kerberos SPNEGO security
related functionality.
Thanks
On Fri, Jul 8, 2016 at 11:28 AM, Pramod Immaneni <pr...@datatorrent.com>
wrote:
> Hi,
>
> We have seen an issue with different versions of
> org.apache.httpcomponents:httpclient jar getting included in classpath when
> apps are launched on newer versions of Hadoop. This is because we have an
> explicit dependency in Apex to version 4.3.5, the version of hadoop we
> build against (2.2 for backward compatibility) does not have a dependency
> to this artifact but newer versions of hadoop have this dependency. To get
> around this problem we are thinking of shading this dependency internally
> in apex. Any other ideas or suggestions?
>
> Thanks
>