Regarding Hanging Tez queries in Nifi

[Varun Rao <va...@gmail.com>]

Hello,

We are using Nifi (1.6.0.3.2.0.8-3) with Hive (1.2), hive is running on tez
(2.6.5.0-292). The query on tez-view shows it is completed but on Nifi we
still see tez queries queued up 10 minutes after the query is finished on
tez-view.  We have considered lowering the parameter
tez.am.container.idle.release-timeout-min.millis however this hasn't made
any differences.

Do you know what can be causing this?

Thanks

RE: Regarding Hanging Tez queries in Nifi

[Shawn Weeks <sw...@weeksconsulting.us>]

Do you see this behavior if your only running a single query once? Or does it happen when you have a bunch of them queued up? Are there other things using your Hive DBCP Connection Pool that might be starving SelectHiveQL for connections? Is your Max Timer Driven Thread Count high enough to support the number of threads your using across all of your workflows, the default is 10 which is a bit low.

If possible sharing a simplified template of your workflow or some pictures would be helpful.

Thanks
Shawn

From: Varun Rao <va...@gmail.com>
Sent: Thursday, April 25, 2019 9:08 AM
To: users@nifi.apache.org
Subject: Re: Regarding Hanging Tez queries in Nifi

The query is doing a validation (performing a count), so only a single row is being returned.

On Thu, Apr 25, 2019 at 10:05 AM Shawn Weeks <sw...@weeksconsulting.us>> wrote:
How many rows does the query return? The actual fetch of rows doesn’t happen until the query is finished and I’ve seen a million rows or so take several minutes to fetch.

Thanks
Shawn

Sent from my iPhone

> On Apr 25, 2019, at 8:37 AM, Varun Rao <va...@gmail.com>> wrote:
>
> Hello,
>
> We are using Nifi (1.6.0.3.2.0.8-3) with Hive (1.2), hive is running on tez (2.6.5.0-292). The query on tez-view shows it is completed but on Nifi we still see tez queries queued up 10 minutes after the query is finished on tez-view.  We have considered lowering the parameter tez.am.container.idle.release-timeout-min.millis however this hasn't made any differences.
>
> Do you know what can be causing this?
>
> Thanks

Re: Regarding Hanging Tez queries in Nifi

[Varun Rao <va...@gmail.com>]

The query is doing a validation (performing a count), so only a single row
is being returned.

On Thu, Apr 25, 2019 at 10:05 AM Shawn Weeks <sw...@weeksconsulting.us>
wrote:

> How many rows does the query return? The actual fetch of rows doesn’t
> happen until the query is finished and I’ve seen a million rows or so take
> several minutes to fetch.
>
> Thanks
> Shawn
>
> Sent from my iPhone
>
> > On Apr 25, 2019, at 8:37 AM, Varun Rao <va...@gmail.com> wrote:
> >
> > Hello,
> >
> > We are using Nifi (1.6.0.3.2.0.8-3) with Hive (1.2), hive is running on
> tez (2.6.5.0-292). The query on tez-view shows it is completed but on Nifi
> we still see tez queries queued up 10 minutes after the query is finished
> on tez-view.  We have considered lowering the parameter
> tez.am.container.idle.release-timeout-min.millis however this hasn't made
> any differences.
> >
> > Do you know what can be causing this?
> >
> > Thanks
>

Re: Regarding Hanging Tez queries in Nifi

[Shawn Weeks <sw...@weeksconsulting.us>]

How many rows does the query return? The actual fetch of rows doesn’t happen until the query is finished and I’ve seen a million rows or so take several minutes to fetch. 

Thanks
Shawn

Sent from my iPhone

> On Apr 25, 2019, at 8:37 AM, Varun Rao <va...@gmail.com> wrote:
> 
> Hello, 
> 
> We are using Nifi (1.6.0.3.2.0.8-3) with Hive (1.2), hive is running on tez (2.6.5.0-292). The query on tez-view shows it is completed but on Nifi we still see tez queries queued up 10 minutes after the query is finished on tez-view.  We have considered lowering the parameter tez.am.container.idle.release-timeout-min.millis however this hasn't made any differences.
> 
> Do you know what can be causing this?
> 
> Thanks