You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by 徐涛 <ha...@gmail.com> on 2018/10/25 14:27:49 UTC
How to tune Hadoop version in flink shaded jar to Hadoop version
actually used?
Hi Experts
When running flink on YARN, from ClusterEntrypoint the system environment info is print out.
One of the info is "Hadoop version: 2.4.1”, I think it is from the flink-shaded-hadoop2 jar. But actually the system Hadoop version is 2.7.2.
I want to know is it OK if the version is different? Is it a best practice to adjust flink Hadoop version to the Hadoop version actually used?
Thanks a lot.
Best
Henry
Re: How to tune Hadoop version in flink shaded jar to Hadoop version
actually used?
Posted by 徐涛 <ha...@gmail.com>.
Hi Hequn & Vino,
Finally I rebuild the Flink by change the “hadoop.version” in the pom file.
Because Flink use maven shaded plugin to shade the Hadoop dependency, this also means I need to rebuild the hadoop shaded jar each time I upgrade Flink version.
Best
Henry
> 在 2018年10月30日，下午12:35，Hequn Cheng <ch...@gmail.com> 写道：
>
> Hi Henry,
>
> You can specify a specific Hadoop version to build against:
> mvn clean install -DskipTests -Dhadoop.version=2.6.1
> More details here[1].
>
> Best, Hequn
>
> [1] https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/building.html#hadoop-versions <https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/building.html#hadoop-versions>
> On Tue, Oct 30, 2018 at 10:02 AM vino yang <yanghua1127@gmail.com <ma...@gmail.com>> wrote:
> Hi Henry,
>
> You just need to change the node of "hadoop.version" in the parent pom file.
>
> Thanks, vino.
>
> 徐涛 <happydexutao@gmail.com <ma...@gmail.com>> 于2018年10月29日周一 下午11:23写道：
> Hi Vino,
> Because I build the project with Maven, maybe I can not use the jars directly download from the web.
> If built with Maven, how can I adjust the Hadoop version with the Hadoop version really used?
> Thanks a lot!!
>
> Best
> Henry
>
>> 在 2018年10月26日，上午10:02，vino yang <yanghua1127@gmail.com <ma...@gmail.com>> 写道：
>>
>> Hi Henry,
>>
>> When running flink on YARN, from ClusterEntrypoint the system environment info is print out.
>> One of the info is "Hadoop version: 2.4.1”, I think it is from the flink-shaded-hadoop2 jar. But actually the system Hadoop version is 2.7.2.
>>
>> I want to know is it OK if the version is different?
>>
>> > I don't think it is OK, because you will use a lower version of the client to access the higher version of the server.
>>
>> Is it a best practice to adjust flink Hadoop version to the Hadoop version actually used?
>>
>> > I personally recommend that you keep the two versions consistent to eliminate the possibility of causing various potential problems.
>> In fact, Flink provides a bundle of Hadoop 2.7.x bundles for you to download.[1]
>>
>> [1]: https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-hadoop27-scala_2.11.tgz <https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-hadoop27-scala_2.11.tgz>
>>
>> Thanks, vino.
>>
>> 徐涛 <happydexutao@gmail.com <ma...@gmail.com>> 于2018年10月26日周五 上午9:13写道：
>> Hi Experts
>> When running flink on YARN, from ClusterEntrypoint the system environment info is print out.
>> One of the info is "Hadoop version: 2.4.1”, I think it is from the flink-shaded-hadoop2 jar. But actually the system Hadoop version is 2.7.2.
>> I want to know is it OK if the version is different? Is it a best practice to adjust flink Hadoop version to the Hadoop version actually used?
>>
>> Thanks a lot.
>>
>> Best
>> Henry
>
Re: How to tune Hadoop version in flink shaded jar to Hadoop version
actually used?
Posted by Hequn Cheng <ch...@gmail.com>.
Hi Henry,
You can specify a specific Hadoop version to build against:
> mvn clean install -DskipTests -Dhadoop.version=2.6.1
More details here[1].
Best, Hequn
[1]
https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/building.html#hadoop-versions
On Tue, Oct 30, 2018 at 10:02 AM vino yang <ya...@gmail.com> wrote:
> Hi Henry,
>
> You just need to change the node of "hadoop.version" in the parent pom
> file.
>
> Thanks, vino.
>
> 徐涛 <ha...@gmail.com> 于2018年10月29日周一 下午11:23写道：
>
>> Hi Vino,
>> Because I build the project with Maven, maybe I can not use the jars
>> directly download from the web.
>> If built with Maven, how can I adjust the Hadoop version with the Hadoop
>> version really used?
>> Thanks a lot!!
>>
>> Best
>> Henry
>>
>> 在 2018年10月26日，上午10:02，vino yang <ya...@gmail.com> 写道：
>>
>> Hi Henry,
>>
>> When running flink on YARN, from ClusterEntrypoint the system environment
>> info is print out.
>> One of the info is "Hadoop version: 2.4.1”, I think it is from the
>> flink-shaded-hadoop2 jar. But actually the system Hadoop version is 2.7.2.
>>
>> I want to know is it OK if the version is different?
>>
>> *> I don't think it is OK, because you will use a lower version of the
>> client to access the higher version of the server.*
>>
>> Is it a best practice to adjust flink Hadoop version to the Hadoop
>> version actually used?
>>
>> *> I personally recommend that you keep the two versions consistent to
>> eliminate the possibility of causing various potential problems. *
>> *In fact, Flink provides a bundle of Hadoop 2.7.x bundles for you to
>> download.[1]*
>>
>> [1]:
>> https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-hadoop27-scala_2.11.tgz
>>
>> Thanks, vino.
>>
>> 徐涛 <ha...@gmail.com> 于2018年10月26日周五 上午9:13写道：
>>
>>> Hi Experts
>>> When running flink on YARN, from ClusterEntrypoint the system
>>> environment info is print out.
>>> One of the info is "Hadoop version: 2.4.1”, I think it is from
>>> the flink-shaded-hadoop2 jar. But actually the system Hadoop version is
>>> 2.7.2.
>>> I want to know is it OK if the version is different? Is it a
>>> best practice to adjust flink Hadoop version to the Hadoop version actually
>>> used?
>>>
>>> Thanks a lot.
>>>
>>> Best
>>> Henry
>>
>>
>>
Re: How to tune Hadoop version in flink shaded jar to Hadoop version
actually used?
Posted by vino yang <ya...@gmail.com>.
Hi Henry,
You just need to change the node of "hadoop.version" in the parent pom file.
Thanks, vino.
徐涛 <ha...@gmail.com> 于2018年10月29日周一 下午11:23写道：
> Hi Vino,
> Because I build the project with Maven, maybe I can not use the jars
> directly download from the web.
> If built with Maven, how can I adjust the Hadoop version with the Hadoop
> version really used?
> Thanks a lot!!
>
> Best
> Henry
>
> 在 2018年10月26日，上午10:02，vino yang <ya...@gmail.com> 写道：
>
> Hi Henry,
>
> When running flink on YARN, from ClusterEntrypoint the system environment
> info is print out.
> One of the info is "Hadoop version: 2.4.1”, I think it is from the
> flink-shaded-hadoop2 jar. But actually the system Hadoop version is 2.7.2.
>
> I want to know is it OK if the version is different?
>
> *> I don't think it is OK, because you will use a lower version of the
> client to access the higher version of the server.*
>
> Is it a best practice to adjust flink Hadoop version to the Hadoop version
> actually used?
>
> *> I personally recommend that you keep the two versions consistent to
> eliminate the possibility of causing various potential problems. *
> *In fact, Flink provides a bundle of Hadoop 2.7.x bundles for you to
> download.[1]*
>
> [1]:
> https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-hadoop27-scala_2.11.tgz
>
> Thanks, vino.
>
> 徐涛 <ha...@gmail.com> 于2018年10月26日周五 上午9:13写道：
>
>> Hi Experts
>> When running flink on YARN, from ClusterEntrypoint the system
>> environment info is print out.
>> One of the info is "Hadoop version: 2.4.1”, I think it is from
>> the flink-shaded-hadoop2 jar. But actually the system Hadoop version is
>> 2.7.2.
>> I want to know is it OK if the version is different? Is it a best
>> practice to adjust flink Hadoop version to the Hadoop version actually used?
>>
>> Thanks a lot.
>>
>> Best
>> Henry
>
>
>
Re: How to tune Hadoop version in flink shaded jar to Hadoop version
actually used?
Posted by 徐涛 <ha...@gmail.com>.
Hi Vino,
Because I build the project with Maven, maybe I can not use the jars directly download from the web.
If built with Maven, how can I adjust the Hadoop version with the Hadoop version really used?
Thanks a lot!!
Best
Henry
> 在 2018年10月26日，上午10:02，vino yang <ya...@gmail.com> 写道：
>
> Hi Henry,
>
> When running flink on YARN, from ClusterEntrypoint the system environment info is print out.
> One of the info is "Hadoop version: 2.4.1”, I think it is from the flink-shaded-hadoop2 jar. But actually the system Hadoop version is 2.7.2.
>
> I want to know is it OK if the version is different?
>
> > I don't think it is OK, because you will use a lower version of the client to access the higher version of the server.
>
> Is it a best practice to adjust flink Hadoop version to the Hadoop version actually used?
>
> > I personally recommend that you keep the two versions consistent to eliminate the possibility of causing various potential problems.
> In fact, Flink provides a bundle of Hadoop 2.7.x bundles for you to download.[1]
>
> [1]: https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-hadoop27-scala_2.11.tgz <https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-hadoop27-scala_2.11.tgz>
>
> Thanks, vino.
>
> 徐涛 <happydexutao@gmail.com <ma...@gmail.com>> 于2018年10月26日周五 上午9:13写道：
> Hi Experts
> When running flink on YARN, from ClusterEntrypoint the system environment info is print out.
> One of the info is "Hadoop version: 2.4.1”, I think it is from the flink-shaded-hadoop2 jar. But actually the system Hadoop version is 2.7.2.
> I want to know is it OK if the version is different? Is it a best practice to adjust flink Hadoop version to the Hadoop version actually used?
>
> Thanks a lot.
>
> Best
> Henry
Re: How to tune Hadoop version in flink shaded jar to Hadoop version
actually used?
Posted by vino yang <ya...@gmail.com>.
Hi Henry,
When running flink on YARN, from ClusterEntrypoint the system environment
info is print out.
One of the info is "Hadoop version: 2.4.1”, I think it is from the
flink-shaded-hadoop2 jar. But actually the system Hadoop version is 2.7.2.
I want to know is it OK if the version is different?
*> I don't think it is OK, because you will use a lower version of the
client to access the higher version of the server.*
Is it a best practice to adjust flink Hadoop version to the Hadoop version
actually used?
*> I personally recommend that you keep the two versions consistent to
eliminate the possibility of causing various potential problems. *
*In fact, Flink provides a bundle of Hadoop 2.7.x bundles for you to
download.[1]*
[1]:
https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-hadoop27-scala_2.11.tgz
Thanks, vino.
徐涛 <ha...@gmail.com> 于2018年10月26日周五 上午9:13写道：
> Hi Experts
> When running flink on YARN, from ClusterEntrypoint the system
> environment info is print out.
> One of the info is "Hadoop version: 2.4.1”, I think it is from the
> flink-shaded-hadoop2 jar. But actually the system Hadoop version is 2.7.2.
> I want to know is it OK if the version is different? Is it a best
> practice to adjust flink Hadoop version to the Hadoop version actually used?
>
> Thanks a lot.
>
> Best
> Henry