You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ambari.apache.org by Brian Jeltema <bd...@gmail.com> on 2015/11/18 17:58:37 UTC

tez failure after upgrade

On completion of a rolling upgrade to HDP 2.3, tez failed it’s service check with:

15/11/18 11:44:49 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED
15/11/18 11:44:49 INFO examples.OrderedWordCount: DAG diagnostics: [Incompatible versions found, clientVersion=0.7.0.2.3.2.0-2950, AMVersion=0.5.2.2.2.8.0-3150]

The clientVersion is the version after the upgrade, but the AMVersion is the version before the upgrade.
Does anyone know what might cause this?

Thanks
Brian

Re: tez failure after upgrade

Posted by Alejandro Fernandez <af...@hortonworks.com>.
That will be fixed in the next version of Ambari.
https://issues.apache.org/jira/browse/AMBARI-13713

From: Brian Jeltema <bd...@gmail.com>>
Reply-To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Date: Wednesday, November 18, 2015 at 10:08 AM
To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Subject: Re: tez failure after upgrade

I was running Ambari 2.1.2.1. The problem was that the value of tez.lib.uris was not updated by the
rolling upgrade. Updating it and restarting the tez clients fixed the problem. Thanks.

Brian

On Nov 18, 2015, at 12:43 PM, Hitesh Shah <hi...@apache.org>> wrote:

Hello Brian,

What version of Ambari are you running? I believe there were bugs in earlier versions that uploaded the wrong version of the Tez tarball to HDFS during an upgrade.

You can check this by comparing the tez tarball on HDFS ( pointed to by tez.lib.uris in tez-site.xml ) to the tez tarball in the respective version of the Tez rpm. This can be easily fixed by deleting the wrong tarball in HDFS under the the 2.3.2 version dir and replacing it with the correct one.

This may also impact the MapReduce tarball too but MR jobs do not fail as they don't have a compatibility check in place.

thanks
- Hitesh

On Nov 18, 2015, at 8:58 AM, Brian Jeltema <bd...@gmail.com>> wrote:

On completion of a rolling upgrade to HDP 2.3, tez failed it's service check with:

15/11/18 11:44:49 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED
15/11/18 11:44:49 INFO examples.OrderedWordCount: DAG diagnostics: [Incompatible versions found, clientVersion=0.7.0.2.3.2.0-2950, AMVersion=0.5.2.2.2.8.0-3150]

The clientVersion is the version after the upgrade, but the AMVersion is the version before the upgrade.
Does anyone know what might cause this?

Thanks
Brian



Re: tez failure after upgrade

Posted by Brian Jeltema <bd...@gmail.com>.
I was running Ambari 2.1.2.1. The problem was that the value of tez.lib.uris was not updated by the
rolling upgrade. Updating it and restarting the tez clients fixed the problem. Thanks.

Brian

> On Nov 18, 2015, at 12:43 PM, Hitesh Shah <hi...@apache.org> wrote:
> 
> Hello Brian, 
> 
> What version of Ambari are you running? I believe there were bugs in earlier versions that uploaded the wrong version of the Tez tarball to HDFS during an upgrade. 
> 
> You can check this by comparing the tez tarball on HDFS ( pointed to by tez.lib.uris in tez-site.xml ) to the tez tarball in the respective version of the Tez rpm. This can be easily fixed by deleting the wrong tarball in HDFS under the the 2.3.2 version dir and replacing it with the correct one. 
> 
> This may also impact the MapReduce tarball too but MR jobs do not fail as they don’t have a compatibility check in place.
> 
> thanks
> — Hitesh
> 
> On Nov 18, 2015, at 8:58 AM, Brian Jeltema <bd...@gmail.com> wrote:
> 
>> On completion of a rolling upgrade to HDP 2.3, tez failed it’s service check with:
>> 
>> 15/11/18 11:44:49 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED
>> 15/11/18 11:44:49 INFO examples.OrderedWordCount: DAG diagnostics: [Incompatible versions found, clientVersion=0.7.0.2.3.2.0-2950, AMVersion=0.5.2.2.2.8.0-3150]
>> 
>> The clientVersion is the version after the upgrade, but the AMVersion is the version before the upgrade.
>> Does anyone know what might cause this?
>> 
>> Thanks
>> Brian
> 


Re: tez failure after upgrade

Posted by Hitesh Shah <hi...@apache.org>.
Hello Brian, 

What version of Ambari are you running? I believe there were bugs in earlier versions that uploaded the wrong version of the Tez tarball to HDFS during an upgrade. 

You can check this by comparing the tez tarball on HDFS ( pointed to by tez.lib.uris in tez-site.xml ) to the tez tarball in the respective version of the Tez rpm. This can be easily fixed by deleting the wrong tarball in HDFS under the the 2.3.2 version dir and replacing it with the correct one. 

This may also impact the MapReduce tarball too but MR jobs do not fail as they don’t have a compatibility check in place.

thanks
— Hitesh

On Nov 18, 2015, at 8:58 AM, Brian Jeltema <bd...@gmail.com> wrote:

> On completion of a rolling upgrade to HDP 2.3, tez failed it’s service check with:
> 
> 15/11/18 11:44:49 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED
> 15/11/18 11:44:49 INFO examples.OrderedWordCount: DAG diagnostics: [Incompatible versions found, clientVersion=0.7.0.2.3.2.0-2950, AMVersion=0.5.2.2.2.8.0-3150]
> 
> The clientVersion is the version after the upgrade, but the AMVersion is the version before the upgrade.
> Does anyone know what might cause this?
> 
> Thanks
> Brian