hive on tez strange issue

[刘家财 <ji...@gmail.com>]

I currently use tez to speed hive exec time.
To be clarify first, I am using apache hadoop 2.6.0 (configured with
fair_scheduler), tez 0.7.0 built from github repo running on CentOS release
6.6 (Final) * 16.

After running some days, a strange issue came up, yarn can accept
newly-submitted application, but leave them PENDING. and from webUI page,I
am see following

​It is strange to me that there is negative number, and I guess this issue
belong to either tez or fair-scheduler,or something happend only when mixed
them two.

I am sorry I can't reproduce this issue for it happen at intervals.And
there are no exceptions in resourcemanager log, just kinds of audit logs.

Let me know if you need more info to hack on this issue.

Re: hive on tez strange issue

[Hitesh Shah <hi...@apache.org>]

I think this might be a bug somewhere in YARN ( specifically the fair scheduler layer ) as Tez has no control over the data being displayed. The resource usage and accounting is all handled within the ResourceManager in YARN. 

You should probably file a jira against YARN for this issue and attach the screenshot as well as the ResourceManager logs.

A couple of things to check:
   - do you have fair.SchedulingPolicy set to FairShare or DominantResourceFairness ? 
   - In earlier versions, I did notice negative numbers but those were usually when DominantResourceFairness was in play. Not sure if these issues are still open.

thanks
— Hitesh


[1] - https://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html

On Jul 15, 2015, at 8:24 PM, 刘家财 <ji...@gmail.com> wrote:

> I currently use tez to speed hive exec time.
> To be clarify first, I am using apache hadoop 2.6.0 (configured with fair_scheduler), tez 0.7.0 built from github repo running on CentOS release 6.6 (Final) * 16.
> 
> After running some days, a strange issue came up, yarn can accept newly-submitted application, but leave them PENDING. and from webUI page,I am see following
> <Snip20150714_4.png>
> ​It is strange to me that there is negative number, and I guess this issue belong to either tez or fair-scheduler,or something happend only when mixed them two.
> 
> I am sorry I can't reproduce this issue for it happen at intervals.And there are no exceptions in resourcemanager log, just kinds of audit logs.
> 
> Let me know if you need more info to hack on this issue.