Regarding Hanging Tez Queries in Nifi

[Varun Rao <va...@gmail.com>]

Hello,

We are using Nifi (1.6.0.3.2.0.8-3) with Hive (1.2), hive is running on tez
(2.6.5.0-292). The query on tez-view shows it is completed but on Nifi we
still see tez queries queued up 10 minutes after the query is finished on
tez-view.  We have considered lowering the parameter
tez.am.container.idle.release-timeout-min.millis however this hasn't made
any differences.

Do you know what can be causing this?

Thanks

Re: Regarding Hanging Tez Queries in Nifi

[Jonathan Eagles <je...@gmail.com>]

Varun, Thanks for reaching out about this issue. With no experience
direct experience with NiFi, my help may not be fully accurate.
However, here is what could be going on. Hive queries can launch Tez
in session mode and non-session mode. Non-session mode execution is
simple 1) launch an AM (application master controlling process) 2)
submit one query (as a DAG) 3) after completion shutdown the AM. In
this, one can see that a query (and hence one DAG) is executed. In
session mode, The AM stays up to allow for submission of multiple
queries (DAGs). So the session execution is more complex 1) launch an
AM 2) 2) Submit one query 3) Repeat step until session timeout is
expired.

It could be that NiFi is launching the Hive query in session mode with
a long timeout and not shutting down the AM. In that case the below
steps may help.

The configuration to control session mode timeout is below. The
default is "300" meaning 300 seconds or 5 minutes.
tez.session.am.dag.submit.timeout.secs

It could also be that NiFi is unable to accurately track the status of
Tez jobs' completion, in that case I would reach out to NiFi team.

As we get closer to solving this. I may ask you at some point for logs
if the above solution doesn't work.
Regards,
jeagles
Apache Tez PMC Chair

On Thu, Apr 25, 2019 at 8:54 AM Varun Rao <va...@gmail.com> wrote:
>
> Hello,
>
> We are using Nifi (1.6.0.3.2.0.8-3) with Hive (1.2), hive is running on tez (2.6.5.0-292). The query on tez-view shows it is completed but on Nifi we still see tez queries queued up 10 minutes after the query is finished on tez-view.  We have considered lowering the parameter tez.am.container.idle.release-timeout-min.millis however this hasn't made any differences.
>
> Do you know what can be causing this?
>
> Thanks