Hive 0.13 - Yarn - Tez application not shutting down

[Douglas Moore <do...@thinkbiganalytics.com>]

I'm on HDP 2.1 Hive 0.13/Tez

I run this query:

hive -v --database=ORC_db -e `select * from store_sales where
ss_ticket_number=5741230 and ss_item_sk=4825`

The yarn application runs Tez for a few seconds, generates the results and
shuts down and goes away as reported by `yarn application -list`. All is
well.

I run the query below, Tez starts up, runs for a few seconds, generates the
same results, Tez session shuts down and then the yarn application still
shows up as 'RUNNING' minutes later.
Why? How do I fix this 'leak' or re-use the YARN application?
The problem is after running 30 or so of these type of scripts via hive,
the jobs stop running and 'hang'

Thanks!

hive -v --database=ORC_db -f
set hive.optimize.ppd=true
set hive.exec.parallel=true
set hive.vectorized.execution.enabled=true
set io.sort.mb=512
set hive.exec.reducers.bytes.per.reducer=134217728
set hive.auto.convert.join.noconditionaltask.size=1200000000
set mapreduce.job.reduce.slowstart.completedmaps=0.5
set hive.mapjoin.localtask.max.memory.usage=0.99

select * from store_sales where ss_ticket_number=5741230 and ss_item_sk=4825

Re: Hive 0.13 - Yarn - Tez application not shutting down

[Douglas Moore <do...@thinkbiganalytics.com>]

set hive.exec.parallel=true; will cause the Yarn application to linger
forever.

Set hive.exec.parallel=false and the Yarn application disappears after
running the Hive CLI finishes.

Is this a Hive, Yarn or Tez bug?


On Thu, Jun 12, 2014 at 4:15 PM, Douglas Moore <
douglas.moore@thinkbiganalytics.com> wrote:

> I'm on HDP 2.1 Hive 0.13/Tez
>
> I run this query:
>
> hive -v --database=ORC_db -e `select * from store_sales where
> ss_ticket_number=5741230 and ss_item_sk=4825`
>
> The yarn application runs Tez for a few seconds, generates the results and
> shuts down and goes away as reported by `yarn application -list`. All is
> well.
>
> I run the query below, Tez starts up, runs for a few seconds, generates
> the same results, Tez session shuts down and then the yarn application
> still shows up as 'RUNNING' minutes later.
> Why? How do I fix this 'leak' or re-use the YARN application?
> The problem is after running 30 or so of these type of scripts via hive,
> the jobs stop running and 'hang'
>
> Thanks!
>
> hive -v --database=ORC_db -f
> set hive.optimize.ppd=true
> set hive.exec.parallel=true
> set hive.vectorized.execution.enabled=true
> set io.sort.mb=512
> set hive.exec.reducers.bytes.per.reducer=134217728
> set hive.auto.convert.join.noconditionaltask.size=1200000000
> set mapreduce.job.reduce.slowstart.completedmaps=0.5
> set hive.mapjoin.localtask.max.memory.usage=0.99
>
> select * from store_sales where ss_ticket_number=5741230 and
> ss_item_sk=4825
>



-- 





*Douglas MoorePrincipal ConsultantThink Big
Analyticsdouglas.moore@thinkbiganalytics.com
<do...@thinkbiganalytics.com>m. 781-454-5971 o. 781-209-5565
@douglas_ma*