You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by "Natarajan, Prabakaran 1. (NSN - IN/Bangalore)" <pr...@nsn.com> on 2014/05/19 07:31:06 UTC
Tez - Hive query Application not killed
Hi
If I run a hive query with Tez as engine then in Application Tracker - I see application(or JVM) created for the query execution is not getting killed immediately after the query execution. It takes about 5 minutes to kill the application .
While this application (orJVM) holds some amount of heap memory as well. Also I noted for each query execution new application (or JVM) is getting created. It seems there is no reuse of application/JVM , then why it is consuming time to kill the application/JVM after query execution?
Thanks and Regards
Prabakaran.N aka NP
nsn, Bangalore
When "I" is replaced by "We" - even Illness becomes "Wellness"
Re: Tez - Hive query Application not killed
Posted by Rajesh Balamohan <ra...@gmail.com>.
If its for AM timeout, you might want to setup
"tez.session.am.dag.submit.timeout.secs". Default value is 300 seconds.
~Rajesh.B
On Tue, May 20, 2014 at 10:40 AM, Natarajan, Prabakaran 1. (NSN -
IN/Bangalore) <pr...@nsn.com> wrote:
> Hi Rajesh
>
>
>
> Thanks. We tried this, but this is not working with Hive Server.
>
>
>
> *Thanks and Regards*
>
> Prabakaran.N aka NP
>
> nsn, Bangalore
>
> *When "I" is replaced by "We" - even Illness becomes "Wellness"*
>
>
>
>
>
> *From:* ext Rajesh Balamohan [mailto:rajesh.balamohan@gmail.com]
> *Sent:* Monday, May 19, 2014 2:45 PM
> *To:* user@tez.incubator.apache.org
> *Subject:* Re: Tez - Hive query Application not killed
>
>
>
> "tez.am.container.session.delay-allocation-millis" parameter should help
> in tweaking the timeout. You can try setting to 1000 (1 second)
>
>
>
> ~Rajesh.B
>
>
>
> On Mon, May 19, 2014 at 2:19 PM, Natarajan, Prabakaran 1. (NSN -
> IN/Bangalore) <pr...@nsn.com> wrote:
>
> Hi
>
>
>
> Thanks for the reply.
>
>
>
> How can we configure Tez Session’s internal timeout ?
>
>
>
> *Thanks and Regards*
>
> Prabakaran.N aka NP
>
> nsn, Bangalore
>
> *When "I" is replaced by "We" - even Illness becomes "Wellness"*
>
>
>
>
>
> *From:* ext Bikas Saha [mailto:bikas@hortonworks.com]
> *Sent:* Monday, May 19, 2014 12:12 PM
> *To:* user@tez.incubator.apache.org
> *Subject:* RE: Tez - Hive query Application not killed
>
>
>
> Hive uses Tez sessions to launch jobs. Unless a session is closed, it will
> keep running until its internal idle timeout expires.
>
> Hive should be executing sequence of queries in the same session (app).
> Are you running the queries in parallel using HiveServer?
>
>
>
> It may be that this is an issue in the HiveServer/Hive CLI. You may get a
> better response from the Hive user mailing list.
>
>
>
> Bikas
>
>
>
> *From:* Natarajan, Prabakaran 1. (NSN - IN/Bangalore) [mailto:
> prabakaran.1.natarajan@nsn.com]
> *Sent:* Sunday, May 18, 2014 10:31 PM
> *To:* user@tez.incubator.apache.org
> *Subject:* Tez - Hive query Application not killed
>
>
>
> Hi
>
>
>
> If I run a hive query with Tez as engine then in Application Tracker – I
> see application(or JVM) created for the query execution is not getting
> killed immediately after the query execution. It takes about 5 minutes to
> kill the application .
>
> While this application (orJVM) holds some amount of heap memory as well.
> Also I noted for each query execution new application (or JVM) is getting
> created. It seems there is no reuse of application/JVM , then why it is
> consuming time to kill the application/JVM after query execution?
>
>
>
> *Thanks and Regards*
>
> Prabakaran.N aka NP
>
> nsn, Bangalore
>
> *When "I" is replaced by "We" - even Illness becomes "Wellness"*
>
>
>
>
>
>
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
>
>
>
>
> --
> ~Rajesh.B
>
--
~Rajesh.B
Re: Tez - Hive query Application not killed
Posted by Nitin Pawar <ni...@gmail.com>.
hiveserver 1 or hive server 2 ?
On Tue, May 20, 2014 at 10:40 AM, Natarajan, Prabakaran 1. (NSN -
IN/Bangalore) <pr...@nsn.com> wrote:
> Hi Rajesh
>
>
>
> Thanks. We tried this, but this is not working with Hive Server.
>
>
>
> *Thanks and Regards*
>
> Prabakaran.N aka NP
>
> nsn, Bangalore
>
> *When "I" is replaced by "We" - even Illness becomes "Wellness"*
>
>
>
>
>
> *From:* ext Rajesh Balamohan [mailto:rajesh.balamohan@gmail.com]
> *Sent:* Monday, May 19, 2014 2:45 PM
> *To:* user@tez.incubator.apache.org
> *Subject:* Re: Tez - Hive query Application not killed
>
>
>
> "tez.am.container.session.delay-allocation-millis" parameter should help
> in tweaking the timeout. You can try setting to 1000 (1 second)
>
>
>
> ~Rajesh.B
>
>
>
> On Mon, May 19, 2014 at 2:19 PM, Natarajan, Prabakaran 1. (NSN -
> IN/Bangalore) <pr...@nsn.com> wrote:
>
> Hi
>
>
>
> Thanks for the reply.
>
>
>
> How can we configure Tez Session’s internal timeout ?
>
>
>
> *Thanks and Regards*
>
> Prabakaran.N aka NP
>
> nsn, Bangalore
>
> *When "I" is replaced by "We" - even Illness becomes "Wellness"*
>
>
>
>
>
> *From:* ext Bikas Saha [mailto:bikas@hortonworks.com]
> *Sent:* Monday, May 19, 2014 12:12 PM
> *To:* user@tez.incubator.apache.org
> *Subject:* RE: Tez - Hive query Application not killed
>
>
>
> Hive uses Tez sessions to launch jobs. Unless a session is closed, it will
> keep running until its internal idle timeout expires.
>
> Hive should be executing sequence of queries in the same session (app).
> Are you running the queries in parallel using HiveServer?
>
>
>
> It may be that this is an issue in the HiveServer/Hive CLI. You may get a
> better response from the Hive user mailing list.
>
>
>
> Bikas
>
>
>
> *From:* Natarajan, Prabakaran 1. (NSN - IN/Bangalore) [mailto:
> prabakaran.1.natarajan@nsn.com]
> *Sent:* Sunday, May 18, 2014 10:31 PM
> *To:* user@tez.incubator.apache.org
> *Subject:* Tez - Hive query Application not killed
>
>
>
> Hi
>
>
>
> If I run a hive query with Tez as engine then in Application Tracker – I
> see application(or JVM) created for the query execution is not getting
> killed immediately after the query execution. It takes about 5 minutes to
> kill the application .
>
> While this application (orJVM) holds some amount of heap memory as well.
> Also I noted for each query execution new application (or JVM) is getting
> created. It seems there is no reuse of application/JVM , then why it is
> consuming time to kill the application/JVM after query execution?
>
>
>
> *Thanks and Regards*
>
> Prabakaran.N aka NP
>
> nsn, Bangalore
>
> *When "I" is replaced by "We" - even Illness becomes "Wellness"*
>
>
>
>
>
>
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
>
>
>
>
> --
> ~Rajesh.B
>
--
Nitin Pawar
RE: Tez - Hive query Application not killed
Posted by "Natarajan, Prabakaran 1. (NSN - IN/Bangalore)" <pr...@nsn.com>.
Hi Rajesh
Thanks. We tried this, but this is not working with Hive Server.
Thanks and Regards
Prabakaran.N aka NP
nsn, Bangalore
When "I" is replaced by "We" - even Illness becomes "Wellness"
From: ext Rajesh Balamohan [mailto:rajesh.balamohan@gmail.com]
Sent: Monday, May 19, 2014 2:45 PM
To: user@tez.incubator.apache.org
Subject: Re: Tez - Hive query Application not killed
"tez.am.container.session.delay-allocation-millis" parameter should help in tweaking the timeout. You can try setting to 1000 (1 second)
~Rajesh.B
On Mon, May 19, 2014 at 2:19 PM, Natarajan, Prabakaran 1. (NSN - IN/Bangalore) <pr...@nsn.com>> wrote:
Hi
Thanks for the reply.
How can we configure Tez Session’s internal timeout ?
Thanks and Regards
Prabakaran.N aka NP
nsn, Bangalore
When "I" is replaced by "We" - even Illness becomes "Wellness"
From: ext Bikas Saha [mailto:bikas@hortonworks.com<ma...@hortonworks.com>]
Sent: Monday, May 19, 2014 12:12 PM
To: user@tez.incubator.apache.org<ma...@tez.incubator.apache.org>
Subject: RE: Tez - Hive query Application not killed
Hive uses Tez sessions to launch jobs. Unless a session is closed, it will keep running until its internal idle timeout expires.
Hive should be executing sequence of queries in the same session (app). Are you running the queries in parallel using HiveServer?
It may be that this is an issue in the HiveServer/Hive CLI. You may get a better response from the Hive user mailing list.
Bikas
From: Natarajan, Prabakaran 1. (NSN - IN/Bangalore) [mailto:prabakaran.1.natarajan@nsn.com<ma...@nsn.com>]
Sent: Sunday, May 18, 2014 10:31 PM
To: user@tez.incubator.apache.org<ma...@tez.incubator.apache.org>
Subject: Tez - Hive query Application not killed
Hi
If I run a hive query with Tez as engine then in Application Tracker – I see application(or JVM) created for the query execution is not getting killed immediately after the query execution. It takes about 5 minutes to kill the application .
While this application (orJVM) holds some amount of heap memory as well. Also I noted for each query execution new application (or JVM) is getting created. It seems there is no reuse of application/JVM , then why it is consuming time to kill the application/JVM after query execution?
Thanks and Regards
Prabakaran.N aka NP
nsn, Bangalore
When "I" is replaced by "We" - even Illness becomes "Wellness"
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
--
~Rajesh.B
Re: Tez - Hive query Application not killed
Posted by Rajesh Balamohan <ra...@gmail.com>.
"tez.am.container.session.delay-allocation-millis" parameter should help in
tweaking the timeout. You can try setting to 1000 (1 second)
~Rajesh.B
On Mon, May 19, 2014 at 2:19 PM, Natarajan, Prabakaran 1. (NSN -
IN/Bangalore) <pr...@nsn.com> wrote:
> Hi
>
>
>
> Thanks for the reply.
>
>
>
> How can we configure Tez Session’s internal timeout ?
>
>
>
> *Thanks and Regards*
>
> Prabakaran.N aka NP
>
> nsn, Bangalore
>
> *When "I" is replaced by "We" - even Illness becomes "Wellness"*
>
>
>
>
>
> *From:* ext Bikas Saha [mailto:bikas@hortonworks.com]
> *Sent:* Monday, May 19, 2014 12:12 PM
> *To:* user@tez.incubator.apache.org
> *Subject:* RE: Tez - Hive query Application not killed
>
>
>
> Hive uses Tez sessions to launch jobs. Unless a session is closed, it will
> keep running until its internal idle timeout expires.
>
> Hive should be executing sequence of queries in the same session (app).
> Are you running the queries in parallel using HiveServer?
>
>
>
> It may be that this is an issue in the HiveServer/Hive CLI. You may get a
> better response from the Hive user mailing list.
>
>
>
> Bikas
>
>
>
> *From:* Natarajan, Prabakaran 1. (NSN - IN/Bangalore) [mailto:
> prabakaran.1.natarajan@nsn.com]
> *Sent:* Sunday, May 18, 2014 10:31 PM
> *To:* user@tez.incubator.apache.org
> *Subject:* Tez - Hive query Application not killed
>
>
>
> Hi
>
>
>
> If I run a hive query with Tez as engine then in Application Tracker – I
> see application(or JVM) created for the query execution is not getting
> killed immediately after the query execution. It takes about 5 minutes to
> kill the application .
>
> While this application (orJVM) holds some amount of heap memory as well.
> Also I noted for each query execution new application (or JVM) is getting
> created. It seems there is no reuse of application/JVM , then why it is
> consuming time to kill the application/JVM after query execution?
>
>
>
> *Thanks and Regards*
>
> Prabakaran.N aka NP
>
> nsn, Bangalore
>
> *When "I" is replaced by "We" - even Illness becomes "Wellness"*
>
>
>
>
>
>
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
~Rajesh.B
RE: Tez - Hive query Application not killed
Posted by "Natarajan, Prabakaran 1. (NSN - IN/Bangalore)" <pr...@nsn.com>.
Hi
Thanks for the reply.
How can we configure Tez Session’s internal timeout ?
Thanks and Regards
Prabakaran.N aka NP
nsn, Bangalore
When "I" is replaced by "We" - even Illness becomes "Wellness"
From: ext Bikas Saha [mailto:bikas@hortonworks.com]
Sent: Monday, May 19, 2014 12:12 PM
To: user@tez.incubator.apache.org
Subject: RE: Tez - Hive query Application not killed
Hive uses Tez sessions to launch jobs. Unless a session is closed, it will keep running until its internal idle timeout expires.
Hive should be executing sequence of queries in the same session (app). Are you running the queries in parallel using HiveServer?
It may be that this is an issue in the HiveServer/Hive CLI. You may get a better response from the Hive user mailing list.
Bikas
From: Natarajan, Prabakaran 1. (NSN - IN/Bangalore) [mailto:prabakaran.1.natarajan@nsn.com<ma...@nsn.com>]
Sent: Sunday, May 18, 2014 10:31 PM
To: user@tez.incubator.apache.org<ma...@tez.incubator.apache.org>
Subject: Tez - Hive query Application not killed
Hi
If I run a hive query with Tez as engine then in Application Tracker – I see application(or JVM) created for the query execution is not getting killed immediately after the query execution. It takes about 5 minutes to kill the application .
While this application (orJVM) holds some amount of heap memory as well. Also I noted for each query execution new application (or JVM) is getting created. It seems there is no reuse of application/JVM , then why it is consuming time to kill the application/JVM after query execution?
Thanks and Regards
Prabakaran.N aka NP
nsn, Bangalore
When "I" is replaced by "We" - even Illness becomes "Wellness"
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: Tez - Hive query Application not killed
Posted by Bikas Saha <bi...@hortonworks.com>.
Hive uses Tez sessions to launch jobs. Unless a session is closed, it will
keep running until its internal idle timeout expires.
Hive should be executing sequence of queries in the same session (app). Are
you running the queries in parallel using HiveServer?
It may be that this is an issue in the HiveServer/Hive CLI. You may get a
better response from the Hive user mailing list.
Bikas
*From:* Natarajan, Prabakaran 1. (NSN - IN/Bangalore) [mailto:
prabakaran.1.natarajan@nsn.com]
*Sent:* Sunday, May 18, 2014 10:31 PM
*To:* user@tez.incubator.apache.org
*Subject:* Tez - Hive query Application not killed
Hi
If I run a hive query with Tez as engine then in Application Tracker – I
see application(or JVM) created for the query execution is not getting
killed immediately after the query execution. It takes about 5 minutes to
kill the application .
While this application (orJVM) holds some amount of heap memory as well.
Also I noted for each query execution new application (or JVM) is getting
created. It seems there is no reuse of application/JVM , then why it is
consuming time to kill the application/JVM after query execution?
*Thanks and Regards*
Prabakaran.N aka NP
nsn, Bangalore
*When "I" is replaced by "We" - even Illness becomes "Wellness"*
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.