You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Jonathan Hurley <jh...@hortonworks.com> on 2017/10/16 16:22:22 UTC
Review Request 63032: YARN Service Checks Fails Because of Old
hadoop-client Classpath Entry
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63032/
-----------------------------------------------------------
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
Bugs: AMBARI-22245
https://issues.apache.org/jira/browse/AMBARI-22245
Repository: ambari
Description
-------
STR:
- Install basic YARN on HDP 2.6.0.0 (no Tez)
- Upgrade just YARN to HDP 2.6.0.2
Service checks will fail with the following:
```
17/10/12 17:55:20 FATAL distributedshell.ApplicationMaster: Error running ApplicationMaster
java.lang.NoSuchMethodError: org.apache.hadoop.io.retry.RetryPolicies.retryForeverWithFixedSleep(JLjava/util/concurrent/TimeUnit;)Lorg/apache/hadoop/io/retry/RetryPolicy;
at org.apache.hadoop.yarn.client.RMProxy.createRetryPolicy(RMProxy.java:280)
```
- All YARN daemons and clients are reporting 2.6.0.2
- All YARN daemons have loaded JARs for 2.6.0.2
What is happening here is that the applications being run on YARN are picking up the older hadoop-common JAR file. The method {{retryForeverWithFixedSleep}} did not exist in HDP 2.6.0.0.
It is picking up the older JARs for running applications because of the {{yarn-site.xml}} property:
```
yarn-site.xml- <name>yarn.application.classpath</name>
yarn-site.xml: <value>/etc/hadoop/conf,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*,/usr/hdp/current/ext/hadoop/*</value>
```
Ambari should be parameterizing the {{/usr/hdp/current/hadoop-client}} paths here.
Diffs
-----
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py f21719261a
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/status_params.py c2e9d92a04
ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/yarn-site.xml d0b4bb1f0f
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 5dbcf2da1b
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml b7548eab30
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml d3a096a4fa
Diff: https://reviews.apache.org/r/63032/diff/1/
Testing
-------
Manual
Thanks,
Jonathan Hurley
Re: Review Request 63032: YARN Service Checks Fails Because of Old
hadoop-client Classpath Entry
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63032/#review188175
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On Oct. 16, 2017, 12:22 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63032/
> -----------------------------------------------------------
>
> (Updated Oct. 16, 2017, 12:22 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22245
> https://issues.apache.org/jira/browse/AMBARI-22245
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
> - Install basic YARN on HDP 2.6.0.0 (no Tez)
> - Upgrade just YARN to HDP 2.6.0.2
>
> Service checks will fail with the following:
> ```
> 17/10/12 17:55:20 FATAL distributedshell.ApplicationMaster: Error running ApplicationMaster
> java.lang.NoSuchMethodError: org.apache.hadoop.io.retry.RetryPolicies.retryForeverWithFixedSleep(JLjava/util/concurrent/TimeUnit;)Lorg/apache/hadoop/io/retry/RetryPolicy;
> at org.apache.hadoop.yarn.client.RMProxy.createRetryPolicy(RMProxy.java:280)
> ```
>
> - All YARN daemons and clients are reporting 2.6.0.2
> - All YARN daemons have loaded JARs for 2.6.0.2
>
> What is happening here is that the applications being run on YARN are picking up the older hadoop-common JAR file. The method {{retryForeverWithFixedSleep}} did not exist in HDP 2.6.0.0.
>
> It is picking up the older JARs for running applications because of the {{yarn-site.xml}} property:
> ```
> yarn-site.xml- <name>yarn.application.classpath</name>
> yarn-site.xml: <value>/etc/hadoop/conf,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*,/usr/hdp/current/ext/hadoop/*</value>
> ```
>
> Ambari should be parameterizing the {{/usr/hdp/current/hadoop-client}} paths here.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py f21719261a
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/status_params.py c2e9d92a04
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/yarn-site.xml d0b4bb1f0f
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 5dbcf2da1b
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml b7548eab30
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml d3a096a4fa
>
>
> Diff: https://reviews.apache.org/r/63032/diff/1/
>
>
> Testing
> -------
>
> Manual
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 63032: YARN Service Checks Fails Because of Old
hadoop-client Classpath Entry
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63032/#review188151
-----------------------------------------------------------
Ship it!
Ship It!
- Dmitro Lisnichenko
On Oct. 16, 2017, 7:22 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63032/
> -----------------------------------------------------------
>
> (Updated Oct. 16, 2017, 7:22 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22245
> https://issues.apache.org/jira/browse/AMBARI-22245
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
> - Install basic YARN on HDP 2.6.0.0 (no Tez)
> - Upgrade just YARN to HDP 2.6.0.2
>
> Service checks will fail with the following:
> ```
> 17/10/12 17:55:20 FATAL distributedshell.ApplicationMaster: Error running ApplicationMaster
> java.lang.NoSuchMethodError: org.apache.hadoop.io.retry.RetryPolicies.retryForeverWithFixedSleep(JLjava/util/concurrent/TimeUnit;)Lorg/apache/hadoop/io/retry/RetryPolicy;
> at org.apache.hadoop.yarn.client.RMProxy.createRetryPolicy(RMProxy.java:280)
> ```
>
> - All YARN daemons and clients are reporting 2.6.0.2
> - All YARN daemons have loaded JARs for 2.6.0.2
>
> What is happening here is that the applications being run on YARN are picking up the older hadoop-common JAR file. The method {{retryForeverWithFixedSleep}} did not exist in HDP 2.6.0.0.
>
> It is picking up the older JARs for running applications because of the {{yarn-site.xml}} property:
> ```
> yarn-site.xml- <name>yarn.application.classpath</name>
> yarn-site.xml: <value>/etc/hadoop/conf,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*,/usr/hdp/current/ext/hadoop/*</value>
> ```
>
> Ambari should be parameterizing the {{/usr/hdp/current/hadoop-client}} paths here.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py f21719261a
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/status_params.py c2e9d92a04
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/yarn-site.xml d0b4bb1f0f
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 5dbcf2da1b
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml b7548eab30
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml d3a096a4fa
>
>
> Diff: https://reviews.apache.org/r/63032/diff/1/
>
>
> Testing
> -------
>
> Manual
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 63032: YARN Service Checks Fails Because of Old
hadoop-client Classpath Entry
Posted by Dmytro Grinenko <ha...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63032/#review188152
-----------------------------------------------------------
Ship it!
Ship It!
- Dmytro Grinenko
On Oct. 16, 2017, 4:22 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63032/
> -----------------------------------------------------------
>
> (Updated Oct. 16, 2017, 4:22 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22245
> https://issues.apache.org/jira/browse/AMBARI-22245
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
> - Install basic YARN on HDP 2.6.0.0 (no Tez)
> - Upgrade just YARN to HDP 2.6.0.2
>
> Service checks will fail with the following:
> ```
> 17/10/12 17:55:20 FATAL distributedshell.ApplicationMaster: Error running ApplicationMaster
> java.lang.NoSuchMethodError: org.apache.hadoop.io.retry.RetryPolicies.retryForeverWithFixedSleep(JLjava/util/concurrent/TimeUnit;)Lorg/apache/hadoop/io/retry/RetryPolicy;
> at org.apache.hadoop.yarn.client.RMProxy.createRetryPolicy(RMProxy.java:280)
> ```
>
> - All YARN daemons and clients are reporting 2.6.0.2
> - All YARN daemons have loaded JARs for 2.6.0.2
>
> What is happening here is that the applications being run on YARN are picking up the older hadoop-common JAR file. The method {{retryForeverWithFixedSleep}} did not exist in HDP 2.6.0.0.
>
> It is picking up the older JARs for running applications because of the {{yarn-site.xml}} property:
> ```
> yarn-site.xml- <name>yarn.application.classpath</name>
> yarn-site.xml: <value>/etc/hadoop/conf,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*,/usr/hdp/current/ext/hadoop/*</value>
> ```
>
> Ambari should be parameterizing the {{/usr/hdp/current/hadoop-client}} paths here.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py f21719261a
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/status_params.py c2e9d92a04
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/yarn-site.xml d0b4bb1f0f
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 5dbcf2da1b
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml b7548eab30
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml d3a096a4fa
>
>
> Diff: https://reviews.apache.org/r/63032/diff/1/
>
>
> Testing
> -------
>
> Manual
>
>
> Thanks,
>
> Jonathan Hurley
>
>