You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Foster, Craig" <fo...@amazon.com> on 2017/03/17 05:52:54 UTC
Return of Flink shading problems in 1.2.0
Hi:
A few months ago, I was building Flink and ran into shading issues for flink-dist as described in your docs. We resolved this in BigTop by adding the correct way to build flink-dist in the do-component-build script and everything was fine after that.
Now, I’m running into issues doing the same now in Flink 1.2.0 and I’m trying to figure out what’s changed and how to fix it. Here’s how the flink-dist jar looks with proper shading:
jar -tvf /usr/lib/flink/lib/flink-dist_2.10-1.1.4.jar | grep HttpConnectionParams
2485 Tue Jan 01 00:00:00 UTC 1980 org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
3479 Tue Jan 01 00:00:00 UTC 1980 org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
When I build Flink 1.2.0 in BigTop, here’s shading for the jar found in the RPM:
jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
2392 Tue Jan 01 00:00:00 GMT 1980 org/apache/commons/httpclient/params/HttpConnectionParams.class
2485 Tue Jan 01 00:00:00 GMT 1980 org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
3479 Tue Jan 01 00:00:00 GMT 1980 org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
2868 Tue Jan 01 00:00:00 GMT 1980 org/apache/http/params/HttpConnectionParams.class
I thought maybe it was some strange thing going on with BigTop, so then I tried just straight building Flink 1.2.0 (outside BigTop) and get the same shading:
jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
2485 Fri Mar 17 05:41:16 GMT 2017 org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
3479 Fri Mar 17 05:41:16 GMT 2017 org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
2392 Fri Mar 17 05:41:24 GMT 2017 org/apache/commons/httpclient/params/HttpConnectionParams.class
2868 Fri Mar 17 05:41:24 GMT 2017 org/apache/http/params/HttpConnectionParams.class
And, yes, this is after going into flink-dist and running mvn clean install again since I am using Maven 3.3.x.
Here’s a snippet from my Maven version:
mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00)
Maven home: /usr/local/apache-maven
Java version: 1.8.0_121, vendor: Oracle Corporation
Any ideas on what my problem might be here?
Thanks,
Craig
Re: Return of Flink shading problems in 1.2.0
Posted by Robert Metzger <rm...@apache.org>.
Here is the JIRA: https://issues.apache.org/jira/browse/FLINK-6125
On Mon, Mar 20, 2017 at 10:27 AM, Robert Metzger <rm...@apache.org>
wrote:
> Hi Craig,
>
> I was able to reproduce the issue with maven 3.3 in Flink 1.2. I'll look
> into it.
>
> On Fri, Mar 17, 2017 at 11:56 PM, Foster, Craig <fo...@amazon.com>
> wrote:
>
>> Ping. So I’ve built with 3.0.5 and it does give proper shading. So it
>> does get me yet another workaround where my only recourse is to use a max
>> version of Maven. Still, I feel there should be a long-term fix at some
>> point in time.
>>
>>
>>
>> I also believe there is a regression in Flink 1.2.0 for Maven 3.3.x with
>> the process as documented, so hoping someone can at least duplicate or let
>> me know of a new workaround for 3.3.x.
>>
>>
>>
>> Thanks!
>>
>> Craig
>>
>>
>>
>> *From: *"Foster, Craig" <fo...@amazon.com>
>> *Reply-To: *"user@flink.apache.org" <us...@flink.apache.org>
>> *Date: *Friday, March 17, 2017 at 7:23 AM
>> *To: *"user@flink.apache.org" <us...@flink.apache.org>
>> *Cc: *Ufuk Celebi <uc...@apache.org>, Robert Metzger <rm...@apache.org>,
>> Stephan Ewen <se...@apache.org>
>> *Subject: *Re: Return of Flink shading problems in 1.2.0
>>
>>
>>
>> Hey Stephen:
>>
>> I am building twice in every case described in my previous mail. Well,
>> building then rebuilding the flink-dist submodule.
>>
>>
>>
>> This was fixed in BigTop but I started seeing this issue again with Flink
>> 1.2.0. I was wondering if there's something else in the environment that
>> could prevent the shading from working because it isn't now even with the
>> workaround.
>>
>>
>> On Mar 17, 2017, at 4:08 AM, Stephan Ewen <se...@apache.org> wrote:
>>
>> Hi Craig!
>>
>>
>>
>> Maven 3.3.x has a shading problem. You need to build two times, once from
>> root, once inside "flink-dist". Have a look here:
>>
>>
>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.2/
>> setup/building.html#dependency-shading
>>
>>
>>
>> Maybe that way missed in BigTop?
>>
>>
>>
>> I am wondering if we should actually throw an error if building with
>> Maven 3.3.x - too many users run into that issue.
>>
>>
>>
>> Stephan
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Mar 17, 2017 at 8:14 AM, Ufuk Celebi <uc...@apache.org> wrote:
>>
>> Pulling in Robert and Stephan who know the project's shading setup the
>> best.
>>
>>
>> On Fri, Mar 17, 2017 at 6:52 AM, Foster, Craig <fo...@amazon.com>
>> wrote:
>> > Hi:
>> >
>> > A few months ago, I was building Flink and ran into shading issues for
>> > flink-dist as described in your docs. We resolved this in BigTop by
>> adding
>> > the correct way to build flink-dist in the do-component-build script and
>> > everything was fine after that.
>> >
>> >
>> >
>> > Now, I’m running into issues doing the same now in Flink 1.2.0 and I’m
>> > trying to figure out what’s changed and how to fix it. Here’s how the
>> > flink-dist jar looks with proper shading:
>> >
>> >
>> >
>> > jar -tvf /usr/lib/flink/lib/flink-dist_2.10-1.1.4.jar | grep
>> > HttpConnectionParams
>> > 2485 Tue Jan 01 00:00:00 UTC 1980
>> > org/apache/flink/hadoop/shaded/org/apache/commons/httpclient
>> /params/HttpConnectionParams.class
>> > 3479 Tue Jan 01 00:00:00 UTC 1980
>> > org/apache/flink/hadoop/shaded/org/apache/http/params/HttpCo
>> nnectionParams.class
>> >
>> >
>> >
>> > When I build Flink 1.2.0 in BigTop, here’s shading for the jar found in
>> the
>> > RPM:
>> >
>> >
>> >
>> > jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
>> > 2392 Tue Jan 01 00:00:00 GMT 1980
>> > org/apache/commons/httpclient/params/HttpConnectionParams.class
>> > 2485 Tue Jan 01 00:00:00 GMT 1980
>> > org/apache/flink/hadoop/shaded/org/apache/commons/httpclient
>> /params/HttpConnectionParams.class
>> > 3479 Tue Jan 01 00:00:00 GMT 1980
>> > org/apache/flink/hadoop/shaded/org/apache/http/params/HttpCo
>> nnectionParams.class
>> > 2868 Tue Jan 01 00:00:00 GMT 1980
>> > org/apache/http/params/HttpConnectionParams.class
>> >
>> >
>> >
>> > I thought maybe it was some strange thing going on with BigTop, so then
>> I
>> > tried just straight building Flink 1.2.0 (outside BigTop) and get the
>> same
>> > shading:
>> >
>> >
>> >
>> > jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
>> >
>> > 2485 Fri Mar 17 05:41:16 GMT 2017
>> > org/apache/flink/hadoop/shaded/org/apache/commons/httpclient
>> /params/HttpConnectionParams.class
>> >
>> > 3479 Fri Mar 17 05:41:16 GMT 2017
>> > org/apache/flink/hadoop/shaded/org/apache/http/params/HttpCo
>> nnectionParams.class
>> >
>> > 2392 Fri Mar 17 05:41:24 GMT 2017
>> > org/apache/commons/httpclient/params/HttpConnectionParams.class
>> >
>> > 2868 Fri Mar 17 05:41:24 GMT 2017
>> > org/apache/http/params/HttpConnectionParams.class
>> >
>> >
>> >
>> > And, yes, this is after going into flink-dist and running mvn clean
>> install
>> > again since I am using Maven 3.3.x.
>> >
>> >
>> >
>> > Here’s a snippet from my Maven version:
>> >
>> > mvn -version
>> >
>> > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>> > 2015-11-10T16:41:47+00:00)
>> >
>> > Maven home: /usr/local/apache-maven
>> >
>> > Java version: 1.8.0_121, vendor: Oracle Corporation
>> >
>> >
>> >
>> > Any ideas on what my problem might be here?
>> >
>> >
>> >
>> > Thanks,
>> >
>> > Craig
>> >
>> >
>>
>>
>>
>>
>
Re: Return of Flink shading problems in 1.2.0
Posted by Robert Metzger <rm...@apache.org>.
Hi Craig,
I was able to reproduce the issue with maven 3.3 in Flink 1.2. I'll look
into it.
On Fri, Mar 17, 2017 at 11:56 PM, Foster, Craig <fo...@amazon.com> wrote:
> Ping. So I’ve built with 3.0.5 and it does give proper shading. So it does
> get me yet another workaround where my only recourse is to use a max
> version of Maven. Still, I feel there should be a long-term fix at some
> point in time.
>
>
>
> I also believe there is a regression in Flink 1.2.0 for Maven 3.3.x with
> the process as documented, so hoping someone can at least duplicate or let
> me know of a new workaround for 3.3.x.
>
>
>
> Thanks!
>
> Craig
>
>
>
> *From: *"Foster, Craig" <fo...@amazon.com>
> *Reply-To: *"user@flink.apache.org" <us...@flink.apache.org>
> *Date: *Friday, March 17, 2017 at 7:23 AM
> *To: *"user@flink.apache.org" <us...@flink.apache.org>
> *Cc: *Ufuk Celebi <uc...@apache.org>, Robert Metzger <rm...@apache.org>,
> Stephan Ewen <se...@apache.org>
> *Subject: *Re: Return of Flink shading problems in 1.2.0
>
>
>
> Hey Stephen:
>
> I am building twice in every case described in my previous mail. Well,
> building then rebuilding the flink-dist submodule.
>
>
>
> This was fixed in BigTop but I started seeing this issue again with Flink
> 1.2.0. I was wondering if there's something else in the environment that
> could prevent the shading from working because it isn't now even with the
> workaround.
>
>
> On Mar 17, 2017, at 4:08 AM, Stephan Ewen <se...@apache.org> wrote:
>
> Hi Craig!
>
>
>
> Maven 3.3.x has a shading problem. You need to build two times, once from
> root, once inside "flink-dist". Have a look here:
>
>
>
> https://ci.apache.org/projects/flink/flink-docs-
> release-1.2/setup/building.html#dependency-shading
>
>
>
> Maybe that way missed in BigTop?
>
>
>
> I am wondering if we should actually throw an error if building with Maven
> 3.3.x - too many users run into that issue.
>
>
>
> Stephan
>
>
>
>
>
>
>
> On Fri, Mar 17, 2017 at 8:14 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
> Pulling in Robert and Stephan who know the project's shading setup the
> best.
>
>
> On Fri, Mar 17, 2017 at 6:52 AM, Foster, Craig <fo...@amazon.com>
> wrote:
> > Hi:
> >
> > A few months ago, I was building Flink and ran into shading issues for
> > flink-dist as described in your docs. We resolved this in BigTop by
> adding
> > the correct way to build flink-dist in the do-component-build script and
> > everything was fine after that.
> >
> >
> >
> > Now, I’m running into issues doing the same now in Flink 1.2.0 and I’m
> > trying to figure out what’s changed and how to fix it. Here’s how the
> > flink-dist jar looks with proper shading:
> >
> >
> >
> > jar -tvf /usr/lib/flink/lib/flink-dist_2.10-1.1.4.jar | grep
> > HttpConnectionParams
> > 2485 Tue Jan 01 00:00:00 UTC 1980
> > org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/
> HttpConnectionParams.class
> > 3479 Tue Jan 01 00:00:00 UTC 1980
> > org/apache/flink/hadoop/shaded/org/apache/http/params/
> HttpConnectionParams.class
> >
> >
> >
> > When I build Flink 1.2.0 in BigTop, here’s shading for the jar found in
> the
> > RPM:
> >
> >
> >
> > jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
> > 2392 Tue Jan 01 00:00:00 GMT 1980
> > org/apache/commons/httpclient/params/HttpConnectionParams.class
> > 2485 Tue Jan 01 00:00:00 GMT 1980
> > org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/
> HttpConnectionParams.class
> > 3479 Tue Jan 01 00:00:00 GMT 1980
> > org/apache/flink/hadoop/shaded/org/apache/http/params/
> HttpConnectionParams.class
> > 2868 Tue Jan 01 00:00:00 GMT 1980
> > org/apache/http/params/HttpConnectionParams.class
> >
> >
> >
> > I thought maybe it was some strange thing going on with BigTop, so then I
> > tried just straight building Flink 1.2.0 (outside BigTop) and get the
> same
> > shading:
> >
> >
> >
> > jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
> >
> > 2485 Fri Mar 17 05:41:16 GMT 2017
> > org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/
> HttpConnectionParams.class
> >
> > 3479 Fri Mar 17 05:41:16 GMT 2017
> > org/apache/flink/hadoop/shaded/org/apache/http/params/
> HttpConnectionParams.class
> >
> > 2392 Fri Mar 17 05:41:24 GMT 2017
> > org/apache/commons/httpclient/params/HttpConnectionParams.class
> >
> > 2868 Fri Mar 17 05:41:24 GMT 2017
> > org/apache/http/params/HttpConnectionParams.class
> >
> >
> >
> > And, yes, this is after going into flink-dist and running mvn clean
> install
> > again since I am using Maven 3.3.x.
> >
> >
> >
> > Here’s a snippet from my Maven version:
> >
> > mvn -version
> >
> > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> > 2015-11-10T16:41:47+00:00)
> >
> > Maven home: /usr/local/apache-maven
> >
> > Java version: 1.8.0_121, vendor: Oracle Corporation
> >
> >
> >
> > Any ideas on what my problem might be here?
> >
> >
> >
> > Thanks,
> >
> > Craig
> >
> >
>
>
>
>
Re: Return of Flink shading problems in 1.2.0
Posted by "Foster, Craig" <fo...@amazon.com>.
Ping. So I’ve built with 3.0.5 and it does give proper shading. So it does get me yet another workaround where my only recourse is to use a max version of Maven. Still, I feel there should be a long-term fix at some point in time.
I also believe there is a regression in Flink 1.2.0 for Maven 3.3.x with the process as documented, so hoping someone can at least duplicate or let me know of a new workaround for 3.3.x.
Thanks!
Craig
From: "Foster, Craig" <fo...@amazon.com>
Reply-To: "user@flink.apache.org" <us...@flink.apache.org>
Date: Friday, March 17, 2017 at 7:23 AM
To: "user@flink.apache.org" <us...@flink.apache.org>
Cc: Ufuk Celebi <uc...@apache.org>, Robert Metzger <rm...@apache.org>, Stephan Ewen <se...@apache.org>
Subject: Re: Return of Flink shading problems in 1.2.0
Hey Stephen:
I am building twice in every case described in my previous mail. Well, building then rebuilding the flink-dist submodule.
This was fixed in BigTop but I started seeing this issue again with Flink 1.2.0. I was wondering if there's something else in the environment that could prevent the shading from working because it isn't now even with the workaround.
On Mar 17, 2017, at 4:08 AM, Stephan Ewen <se...@apache.org>> wrote:
Hi Craig!
Maven 3.3.x has a shading problem. You need to build two times, once from root, once inside "flink-dist". Have a look here:
https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/building.html#dependency-shading
Maybe that way missed in BigTop?
I am wondering if we should actually throw an error if building with Maven 3.3.x - too many users run into that issue.
Stephan
On Fri, Mar 17, 2017 at 8:14 AM, Ufuk Celebi <uc...@apache.org>> wrote:
Pulling in Robert and Stephan who know the project's shading setup the best.
On Fri, Mar 17, 2017 at 6:52 AM, Foster, Craig <fo...@amazon.com>> wrote:
> Hi:
>
> A few months ago, I was building Flink and ran into shading issues for
> flink-dist as described in your docs. We resolved this in BigTop by adding
> the correct way to build flink-dist in the do-component-build script and
> everything was fine after that.
>
>
>
> Now, I’m running into issues doing the same now in Flink 1.2.0 and I’m
> trying to figure out what’s changed and how to fix it. Here’s how the
> flink-dist jar looks with proper shading:
>
>
>
> jar -tvf /usr/lib/flink/lib/flink-dist_2.10-1.1.4.jar | grep
> HttpConnectionParams
> 2485 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
>
>
> When I build Flink 1.2.0 in BigTop, here’s shading for the jar found in the
> RPM:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
> 2392 Tue Jan 01 00:00:00 GMT 1980
> org/apache/commons/httpclient/params/HttpConnectionParams.class
> 2485 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
> 2868 Tue Jan 01 00:00:00 GMT 1980
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> I thought maybe it was some strange thing going on with BigTop, so then I
> tried just straight building Flink 1.2.0 (outside BigTop) and get the same
> shading:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
>
> 2485 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
>
> 3479 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
> 2392 Fri Mar 17 05:41:24 GMT 2017
> org/apache/commons/httpclient/params/HttpConnectionParams.class
>
> 2868 Fri Mar 17 05:41:24 GMT 2017
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> And, yes, this is after going into flink-dist and running mvn clean install
> again since I am using Maven 3.3.x.
>
>
>
> Here’s a snippet from my Maven version:
>
> mvn -version
>
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T16:41:47+00:00)
>
> Maven home: /usr/local/apache-maven
>
> Java version: 1.8.0_121, vendor: Oracle Corporation
>
>
>
> Any ideas on what my problem might be here?
>
>
>
> Thanks,
>
> Craig
>
>
Re: Return of Flink shading problems in 1.2.0
Posted by "Foster, Craig" <fo...@amazon.com>.
Hey Stephen:
I am building twice in every case described in my previous mail. Well, building then rebuilding the flink-dist submodule.
This was fixed in BigTop but I started seeing this issue again with Flink 1.2.0. I was wondering if there's something else in the environment that could prevent the shading from working because it isn't now even with the workaround.
On Mar 17, 2017, at 4:08 AM, Stephan Ewen <se...@apache.org>> wrote:
Hi Craig!
Maven 3.3.x has a shading problem. You need to build two times, once from root, once inside "flink-dist". Have a look here:
https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/building.html#dependency-shading
Maybe that way missed in BigTop?
I am wondering if we should actually throw an error if building with Maven 3.3.x - too many users run into that issue.
Stephan
On Fri, Mar 17, 2017 at 8:14 AM, Ufuk Celebi <uc...@apache.org>> wrote:
Pulling in Robert and Stephan who know the project's shading setup the best.
On Fri, Mar 17, 2017 at 6:52 AM, Foster, Craig <fo...@amazon.com>> wrote:
> Hi:
>
> A few months ago, I was building Flink and ran into shading issues for
> flink-dist as described in your docs. We resolved this in BigTop by adding
> the correct way to build flink-dist in the do-component-build script and
> everything was fine after that.
>
>
>
> Now, I'm running into issues doing the same now in Flink 1.2.0 and I'm
> trying to figure out what's changed and how to fix it. Here's how the
> flink-dist jar looks with proper shading:
>
>
>
> jar -tvf /usr/lib/flink/lib/flink-dist_2.10-1.1.4.jar | grep
> HttpConnectionParams
> 2485 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
>
>
> When I build Flink 1.2.0 in BigTop, here's shading for the jar found in the
> RPM:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
> 2392 Tue Jan 01 00:00:00 GMT 1980
> org/apache/commons/httpclient/params/HttpConnectionParams.class
> 2485 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
> 2868 Tue Jan 01 00:00:00 GMT 1980
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> I thought maybe it was some strange thing going on with BigTop, so then I
> tried just straight building Flink 1.2.0 (outside BigTop) and get the same
> shading:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
>
> 2485 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
>
> 3479 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
> 2392 Fri Mar 17 05:41:24 GMT 2017
> org/apache/commons/httpclient/params/HttpConnectionParams.class
>
> 2868 Fri Mar 17 05:41:24 GMT 2017
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> And, yes, this is after going into flink-dist and running mvn clean install
> again since I am using Maven 3.3.x.
>
>
>
> Here's a snippet from my Maven version:
>
> mvn -version
>
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T16:41:47+00:00)
>
> Maven home: /usr/local/apache-maven
>
> Java version: 1.8.0_121, vendor: Oracle Corporation
>
>
>
> Any ideas on what my problem might be here?
>
>
>
> Thanks,
>
> Craig
>
>
Re: Return of Flink shading problems in 1.2.0
Posted by Stephan Ewen <se...@apache.org>.
Hi Craig!
Maven 3.3.x has a shading problem. You need to build two times, once from
root, once inside "flink-dist". Have a look here:
https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/building.html#dependency-shading
Maybe that way missed in BigTop?
I am wondering if we should actually throw an error if building with Maven
3.3.x - too many users run into that issue.
Stephan
On Fri, Mar 17, 2017 at 8:14 AM, Ufuk Celebi <uc...@apache.org> wrote:
> Pulling in Robert and Stephan who know the project's shading setup the
> best.
>
> On Fri, Mar 17, 2017 at 6:52 AM, Foster, Craig <fo...@amazon.com>
> wrote:
> > Hi:
> >
> > A few months ago, I was building Flink and ran into shading issues for
> > flink-dist as described in your docs. We resolved this in BigTop by
> adding
> > the correct way to build flink-dist in the do-component-build script and
> > everything was fine after that.
> >
> >
> >
> > Now, I’m running into issues doing the same now in Flink 1.2.0 and I’m
> > trying to figure out what’s changed and how to fix it. Here’s how the
> > flink-dist jar looks with proper shading:
> >
> >
> >
> > jar -tvf /usr/lib/flink/lib/flink-dist_2.10-1.1.4.jar | grep
> > HttpConnectionParams
> > 2485 Tue Jan 01 00:00:00 UTC 1980
> > org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/
> HttpConnectionParams.class
> > 3479 Tue Jan 01 00:00:00 UTC 1980
> > org/apache/flink/hadoop/shaded/org/apache/http/params/
> HttpConnectionParams.class
> >
> >
> >
> > When I build Flink 1.2.0 in BigTop, here’s shading for the jar found in
> the
> > RPM:
> >
> >
> >
> > jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
> > 2392 Tue Jan 01 00:00:00 GMT 1980
> > org/apache/commons/httpclient/params/HttpConnectionParams.class
> > 2485 Tue Jan 01 00:00:00 GMT 1980
> > org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/
> HttpConnectionParams.class
> > 3479 Tue Jan 01 00:00:00 GMT 1980
> > org/apache/flink/hadoop/shaded/org/apache/http/params/
> HttpConnectionParams.class
> > 2868 Tue Jan 01 00:00:00 GMT 1980
> > org/apache/http/params/HttpConnectionParams.class
> >
> >
> >
> > I thought maybe it was some strange thing going on with BigTop, so then I
> > tried just straight building Flink 1.2.0 (outside BigTop) and get the
> same
> > shading:
> >
> >
> >
> > jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
> >
> > 2485 Fri Mar 17 05:41:16 GMT 2017
> > org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/
> HttpConnectionParams.class
> >
> > 3479 Fri Mar 17 05:41:16 GMT 2017
> > org/apache/flink/hadoop/shaded/org/apache/http/params/
> HttpConnectionParams.class
> >
> > 2392 Fri Mar 17 05:41:24 GMT 2017
> > org/apache/commons/httpclient/params/HttpConnectionParams.class
> >
> > 2868 Fri Mar 17 05:41:24 GMT 2017
> > org/apache/http/params/HttpConnectionParams.class
> >
> >
> >
> > And, yes, this is after going into flink-dist and running mvn clean
> install
> > again since I am using Maven 3.3.x.
> >
> >
> >
> > Here’s a snippet from my Maven version:
> >
> > mvn -version
> >
> > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> > 2015-11-10T16:41:47+00:00)
> >
> > Maven home: /usr/local/apache-maven
> >
> > Java version: 1.8.0_121, vendor: Oracle Corporation
> >
> >
> >
> > Any ideas on what my problem might be here?
> >
> >
> >
> > Thanks,
> >
> > Craig
> >
> >
>
Re: Return of Flink shading problems in 1.2.0
Posted by Ufuk Celebi <uc...@apache.org>.
Pulling in Robert and Stephan who know the project's shading setup the best.
On Fri, Mar 17, 2017 at 6:52 AM, Foster, Craig <fo...@amazon.com> wrote:
> Hi:
>
> A few months ago, I was building Flink and ran into shading issues for
> flink-dist as described in your docs. We resolved this in BigTop by adding
> the correct way to build flink-dist in the do-component-build script and
> everything was fine after that.
>
>
>
> Now, I’m running into issues doing the same now in Flink 1.2.0 and I’m
> trying to figure out what’s changed and how to fix it. Here’s how the
> flink-dist jar looks with proper shading:
>
>
>
> jar -tvf /usr/lib/flink/lib/flink-dist_2.10-1.1.4.jar | grep
> HttpConnectionParams
> 2485 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
>
>
> When I build Flink 1.2.0 in BigTop, here’s shading for the jar found in the
> RPM:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
> 2392 Tue Jan 01 00:00:00 GMT 1980
> org/apache/commons/httpclient/params/HttpConnectionParams.class
> 2485 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
> 2868 Tue Jan 01 00:00:00 GMT 1980
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> I thought maybe it was some strange thing going on with BigTop, so then I
> tried just straight building Flink 1.2.0 (outside BigTop) and get the same
> shading:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
>
> 2485 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
>
> 3479 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
> 2392 Fri Mar 17 05:41:24 GMT 2017
> org/apache/commons/httpclient/params/HttpConnectionParams.class
>
> 2868 Fri Mar 17 05:41:24 GMT 2017
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> And, yes, this is after going into flink-dist and running mvn clean install
> again since I am using Maven 3.3.x.
>
>
>
> Here’s a snippet from my Maven version:
>
> mvn -version
>
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T16:41:47+00:00)
>
> Maven home: /usr/local/apache-maven
>
> Java version: 1.8.0_121, vendor: Oracle Corporation
>
>
>
> Any ideas on what my problem might be here?
>
>
>
> Thanks,
>
> Craig
>
>