You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Loïc Chanel <lo...@telecomnancy.net> on 2015/07/29 14:52:24 UTC
Computation timeout
Hi all,
As I'm trying to build a secured and multi-tenant Hadoop cluster with Hive,
I am desperately trying to set a timeout to Hive requests.
My idea is that some users can make mistakes such as a join with wrong
keys, and therefore start an infinite loop believing that they are just
launching a very heavy job. Therefore, I'd like to set a limit to the time
a request should take, in order to kill the job automatically if it exceeds
it.
As such a notion cannot be set directly in YARN, I saw that MapReduce2
provides with its own native timeout property, and I would like to know if
Hive provides with the same property someway.
Did anyone heard about such a thing ?
Thanks in advance for your help,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
Re: Computation timeout
Posted by Lefty Leverenz <le...@gmail.com>.
Hmm, if you did see it somewhere please let us know.
I verified the defaults in copies of HiveConf.java for releases up to 1.1.0:
HiveConf-branches> grep 'hive.server2.session.check.interval' *
branch14-HiveConf.java:
HIVE_SERVER2_SESSION_CHECK_INTERVAL("hive.server2.session.check.interval",
"0ms",
brnch1.0-HiveConf.java:
HIVE_SERVER2_SESSION_CHECK_INTERVAL("hive.server2.session.check.interval",
"0ms",
brnch1.1-HiveConf.java:
HIVE_SERVER2_SESSION_CHECK_INTERVAL("hive.server2.session.check.interval",
"0ms",
HiveConf-branches> grep 'hive.server2.idle.operation.timeout' *
branch14-HiveConf.java:
HIVE_SERVER2_IDLE_OPERATION_TIMEOUT("hive.server2.idle.operation.timeout",
"0ms",
brnch1.0-HiveConf.java:
HIVE_SERVER2_IDLE_OPERATION_TIMEOUT("hive.server2.idle.operation.timeout",
"0ms",
brnch1.1-HiveConf.java:
HIVE_SERVER2_IDLE_OPERATION_TIMEOUT("hive.server2.idle.operation.timeout",
"0ms",
-- Lefty
On Thu, Jul 30, 2015 at 3:53 AM, Loïc Chanel <lo...@telecomnancy.net>
wrote:
> Indeed, I was checking this out on the exact same page, but I'm almost
> convinced that I saw on a documentation that the default value was 3000 for
> the check.interval.
> As I can't find it again, let's say I was tired and my eyes betrayed me.
>
> Thanks a lot,
>
>
> Loïc
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-07-30 9:46 GMT+02:00 Lefty Leverenz <le...@gmail.com>:
>
>> You're right about the typos, but both parameters have defaults of 0 ms:
>>
>> - hive.server2.session.check.interval
>> <https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.session.check.interval>
>> - hive.server2.idle.operation.timeout
>> <https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.idle.operation.timeout>
>>
>>
>> -- Lefty
>>
>> On Thu, Jul 30, 2015 at 3:31 AM, Loïc Chanel <
>> loic.chanel@telecomnancy.net> wrote:
>>
>>> Rats, I think I just figured it out.
>>> #2 Is NEGATIVE 3000, right ? I set it to positive yesterday.
>>> As for #1, I think it is the default value, so I am not sure I have to
>>> set it.
>>>
>>> Can you confirm that there is a typo on the name of your properties
>>> (missing last letter) and that is not the actual name of the properties ?
>>>
>>> I'll try again and keep you informed
>>>
>>>
>>> Loïc CHANEL
>>> Engineering student at TELECOM Nancy
>>> Trainee at Worldline - Villeurbanne
>>>
>>> 2015-07-29 20:15 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>>
>>>> this works for me:
>>>> In hive-site.xml:
>>>> 1. hive.server2.session.check.interva=3000;
>>>> 2. hive.server2.idle.operation.timeou=-30000;
>>>> restart HiveServer2.
>>>>
>>>> at beeline, I do "analyze table X compute statistics for columns",
>>>> which takes longer than 30s. it was aborted by HS2 because of above
>>>> settings. I guess it didn't work for you because you didn't have #1.
>>>>
>>>> --Xuefu
>>>>
>>>> On Wed, Jul 29, 2015 at 9:23 AM, Loïc Chanel <
>>>> loic.chanel@telecomnancy.net> wrote:
>>>>
>>>>> I don't think your solution works, as after more than 4 minutes I
>>>>> could still see logs of my job showing that it was running.
>>>>> Do you have a way to check that even if the job was running, it was
>>>>> not being killed by Hive ?
>>>>> Or another solution ?
>>>>>
>>>>> Thanks for your help,
>>>>>
>>>>>
>>>>> Loïc
>>>>>
>>>>> Loïc CHANEL
>>>>> Engineering student at TELECOM Nancy
>>>>> Trainee at Worldline - Villeurbanne
>>>>>
>>>>> 2015-07-29 16:26 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>>>>>
>>>>>> Yes, I set it to negative 60.
>>>>>>
>>>>>> It's not a problem if the session is killed. That's actually what I
>>>>>> try to do, because I can't allow to a user to try to end an infinite
>>>>>> request.
>>>>>> Therefore I'll try your solution :)
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>
>>>>>> Loïc
>>>>>>
>>>>>> Loïc CHANEL
>>>>>> Engineering student at TELECOM Nancy
>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>
>>>>>> 2015-07-29 16:14 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>>>>>
>>>>>>> Okay. To confirm, you set it to negative 60s?
>>>>>>>
>>>>>>> The next thing you can try is to set
>>>>>>> hive.server2.idle.session.timeou=60000 (60sec) and
>>>>>>> hive.server2.idle.session.check.operation=false. I'm pretty sure this
>>>>>>> works, but the user's session will be killed though.
>>>>>>>
>>>>>>> --Xuefu
>>>>>>>
>>>>>>> On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel <
>>>>>>> loic.chanel@telecomnancy.net> wrote:
>>>>>>>
>>>>>>>> I confirm : I just tried hive.server2.idle.operation.timeout
>>>>>>>> setting it to -60 (seconds), but my veeeeeery slow job have not been
>>>>>>>> killed. The issue here is "what if another user come and try to submit a
>>>>>>>> MapReduce job but the cluster is stuck in an infinite loop ?".
>>>>>>>>
>>>>>>>> Do you or anyone else have another idea ?
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>>
>>>>>>>> Loïc
>>>>>>>>
>>>>>>>> Loïc CHANEL
>>>>>>>> Engineering student at TELECOM Nancy
>>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>>
>>>>>>>> 2015-07-29 15:34 GMT+02:00 Loïc Chanel <
>>>>>>>> loic.chanel@telecomnancy.net>:
>>>>>>>>
>>>>>>>>> No, because I thought the idea of infinite operation was not very
>>>>>>>>> compatible with the "idle" word (as the operation will not stop running),
>>>>>>>>> but I'll try :-)
>>>>>>>>> Thanks for the idea,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Loïc
>>>>>>>>>
>>>>>>>>> Loïc CHANEL
>>>>>>>>> Engineering student at TELECOM Nancy
>>>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>>>
>>>>>>>>> 2015-07-29 15:27 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>>>>>>>>
>>>>>>>>>> Have you tried hive.server2.idle.operation.timeout?
>>>>>>>>>>
>>>>>>>>>> --Xuefu
>>>>>>>>>>
>>>>>>>>>> On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel <
>>>>>>>>>> loic.chanel@telecomnancy.net> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> As I'm trying to build a secured and multi-tenant Hadoop cluster
>>>>>>>>>>> with Hive, I am desperately trying to set a timeout to Hive requests.
>>>>>>>>>>> My idea is that some users can make mistakes such as a join with
>>>>>>>>>>> wrong keys, and therefore start an infinite loop believing that they are
>>>>>>>>>>> just launching a very heavy job. Therefore, I'd like to set a limit to the
>>>>>>>>>>> time a request should take, in order to kill the job automatically if it
>>>>>>>>>>> exceeds it.
>>>>>>>>>>>
>>>>>>>>>>> As such a notion cannot be set directly in YARN, I saw that
>>>>>>>>>>> MapReduce2 provides with its own native timeout property, and I would like
>>>>>>>>>>> to know if Hive provides with the same property someway.
>>>>>>>>>>>
>>>>>>>>>>> Did anyone heard about such a thing ?
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance for your help,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Loïc
>>>>>>>>>>>
>>>>>>>>>>> Loïc CHANEL
>>>>>>>>>>> Engineering student at TELECOM Nancy
>>>>>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Computation timeout
Posted by Loïc Chanel <lo...@telecomnancy.net>.
Indeed, I was checking this out on the exact same page, but I'm almost
convinced that I saw on a documentation that the default value was 3000 for
the check.interval.
As I can't find it again, let's say I was tired and my eyes betrayed me.
Thanks a lot,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-30 9:46 GMT+02:00 Lefty Leverenz <le...@gmail.com>:
> You're right about the typos, but both parameters have defaults of 0 ms:
>
> - hive.server2.session.check.interval
> <https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.session.check.interval>
> - hive.server2.idle.operation.timeout
> <https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.idle.operation.timeout>
>
>
> -- Lefty
>
> On Thu, Jul 30, 2015 at 3:31 AM, Loïc Chanel <loic.chanel@telecomnancy.net
> > wrote:
>
>> Rats, I think I just figured it out.
>> #2 Is NEGATIVE 3000, right ? I set it to positive yesterday.
>> As for #1, I think it is the default value, so I am not sure I have to
>> set it.
>>
>> Can you confirm that there is a typo on the name of your properties
>> (missing last letter) and that is not the actual name of the properties ?
>>
>> I'll try again and keep you informed
>>
>>
>> Loïc CHANEL
>> Engineering student at TELECOM Nancy
>> Trainee at Worldline - Villeurbanne
>>
>> 2015-07-29 20:15 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>
>>> this works for me:
>>> In hive-site.xml:
>>> 1. hive.server2.session.check.interva=3000;
>>> 2. hive.server2.idle.operation.timeou=-30000;
>>> restart HiveServer2.
>>>
>>> at beeline, I do "analyze table X compute statistics for columns", which
>>> takes longer than 30s. it was aborted by HS2 because of above settings. I
>>> guess it didn't work for you because you didn't have #1.
>>>
>>> --Xuefu
>>>
>>> On Wed, Jul 29, 2015 at 9:23 AM, Loïc Chanel <
>>> loic.chanel@telecomnancy.net> wrote:
>>>
>>>> I don't think your solution works, as after more than 4 minutes I could
>>>> still see logs of my job showing that it was running.
>>>> Do you have a way to check that even if the job was running, it was not
>>>> being killed by Hive ?
>>>> Or another solution ?
>>>>
>>>> Thanks for your help,
>>>>
>>>>
>>>> Loïc
>>>>
>>>> Loïc CHANEL
>>>> Engineering student at TELECOM Nancy
>>>> Trainee at Worldline - Villeurbanne
>>>>
>>>> 2015-07-29 16:26 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>>>>
>>>>> Yes, I set it to negative 60.
>>>>>
>>>>> It's not a problem if the session is killed. That's actually what I
>>>>> try to do, because I can't allow to a user to try to end an infinite
>>>>> request.
>>>>> Therefore I'll try your solution :)
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>> Loïc
>>>>>
>>>>> Loïc CHANEL
>>>>> Engineering student at TELECOM Nancy
>>>>> Trainee at Worldline - Villeurbanne
>>>>>
>>>>> 2015-07-29 16:14 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>>>>
>>>>>> Okay. To confirm, you set it to negative 60s?
>>>>>>
>>>>>> The next thing you can try is to set
>>>>>> hive.server2.idle.session.timeou=60000 (60sec) and
>>>>>> hive.server2.idle.session.check.operation=false. I'm pretty sure this
>>>>>> works, but the user's session will be killed though.
>>>>>>
>>>>>> --Xuefu
>>>>>>
>>>>>> On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel <
>>>>>> loic.chanel@telecomnancy.net> wrote:
>>>>>>
>>>>>>> I confirm : I just tried hive.server2.idle.operation.timeout setting
>>>>>>> it to -60 (seconds), but my veeeeeery slow job have not been killed. The
>>>>>>> issue here is "what if another user come and try to submit a MapReduce job
>>>>>>> but the cluster is stuck in an infinite loop ?".
>>>>>>>
>>>>>>> Do you or anyone else have another idea ?
>>>>>>> Thanks,
>>>>>>>
>>>>>>>
>>>>>>> Loïc
>>>>>>>
>>>>>>> Loïc CHANEL
>>>>>>> Engineering student at TELECOM Nancy
>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>
>>>>>>> 2015-07-29 15:34 GMT+02:00 Loïc Chanel <loic.chanel@telecomnancy.net
>>>>>>> >:
>>>>>>>
>>>>>>>> No, because I thought the idea of infinite operation was not very
>>>>>>>> compatible with the "idle" word (as the operation will not stop running),
>>>>>>>> but I'll try :-)
>>>>>>>> Thanks for the idea,
>>>>>>>>
>>>>>>>>
>>>>>>>> Loïc
>>>>>>>>
>>>>>>>> Loïc CHANEL
>>>>>>>> Engineering student at TELECOM Nancy
>>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>>
>>>>>>>> 2015-07-29 15:27 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>>>>>>>
>>>>>>>>> Have you tried hive.server2.idle.operation.timeout?
>>>>>>>>>
>>>>>>>>> --Xuefu
>>>>>>>>>
>>>>>>>>> On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel <
>>>>>>>>> loic.chanel@telecomnancy.net> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> As I'm trying to build a secured and multi-tenant Hadoop cluster
>>>>>>>>>> with Hive, I am desperately trying to set a timeout to Hive requests.
>>>>>>>>>> My idea is that some users can make mistakes such as a join with
>>>>>>>>>> wrong keys, and therefore start an infinite loop believing that they are
>>>>>>>>>> just launching a very heavy job. Therefore, I'd like to set a limit to the
>>>>>>>>>> time a request should take, in order to kill the job automatically if it
>>>>>>>>>> exceeds it.
>>>>>>>>>>
>>>>>>>>>> As such a notion cannot be set directly in YARN, I saw that
>>>>>>>>>> MapReduce2 provides with its own native timeout property, and I would like
>>>>>>>>>> to know if Hive provides with the same property someway.
>>>>>>>>>>
>>>>>>>>>> Did anyone heard about such a thing ?
>>>>>>>>>>
>>>>>>>>>> Thanks in advance for your help,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Loïc
>>>>>>>>>>
>>>>>>>>>> Loïc CHANEL
>>>>>>>>>> Engineering student at TELECOM Nancy
>>>>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Computation timeout
Posted by Lefty Leverenz <le...@gmail.com>.
You're right about the typos, but both parameters have defaults of 0 ms:
- hive.server2.session.check.interval
<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.session.check.interval>
- hive.server2.idle.operation.timeout
<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.idle.operation.timeout>
-- Lefty
On Thu, Jul 30, 2015 at 3:31 AM, Loïc Chanel <lo...@telecomnancy.net>
wrote:
> Rats, I think I just figured it out.
> #2 Is NEGATIVE 3000, right ? I set it to positive yesterday.
> As for #1, I think it is the default value, so I am not sure I have to set
> it.
>
> Can you confirm that there is a typo on the name of your properties
> (missing last letter) and that is not the actual name of the properties ?
>
> I'll try again and keep you informed
>
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-07-29 20:15 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>
>> this works for me:
>> In hive-site.xml:
>> 1. hive.server2.session.check.interva=3000;
>> 2. hive.server2.idle.operation.timeou=-30000;
>> restart HiveServer2.
>>
>> at beeline, I do "analyze table X compute statistics for columns", which
>> takes longer than 30s. it was aborted by HS2 because of above settings. I
>> guess it didn't work for you because you didn't have #1.
>>
>> --Xuefu
>>
>> On Wed, Jul 29, 2015 at 9:23 AM, Loïc Chanel <
>> loic.chanel@telecomnancy.net> wrote:
>>
>>> I don't think your solution works, as after more than 4 minutes I could
>>> still see logs of my job showing that it was running.
>>> Do you have a way to check that even if the job was running, it was not
>>> being killed by Hive ?
>>> Or another solution ?
>>>
>>> Thanks for your help,
>>>
>>>
>>> Loïc
>>>
>>> Loïc CHANEL
>>> Engineering student at TELECOM Nancy
>>> Trainee at Worldline - Villeurbanne
>>>
>>> 2015-07-29 16:26 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>>>
>>>> Yes, I set it to negative 60.
>>>>
>>>> It's not a problem if the session is killed. That's actually what I try
>>>> to do, because I can't allow to a user to try to end an infinite request.
>>>> Therefore I'll try your solution :)
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> Loïc
>>>>
>>>> Loïc CHANEL
>>>> Engineering student at TELECOM Nancy
>>>> Trainee at Worldline - Villeurbanne
>>>>
>>>> 2015-07-29 16:14 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>>>
>>>>> Okay. To confirm, you set it to negative 60s?
>>>>>
>>>>> The next thing you can try is to set
>>>>> hive.server2.idle.session.timeou=60000 (60sec) and
>>>>> hive.server2.idle.session.check.operation=false. I'm pretty sure this
>>>>> works, but the user's session will be killed though.
>>>>>
>>>>> --Xuefu
>>>>>
>>>>> On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel <
>>>>> loic.chanel@telecomnancy.net> wrote:
>>>>>
>>>>>> I confirm : I just tried hive.server2.idle.operation.timeout setting
>>>>>> it to -60 (seconds), but my veeeeeery slow job have not been killed. The
>>>>>> issue here is "what if another user come and try to submit a MapReduce job
>>>>>> but the cluster is stuck in an infinite loop ?".
>>>>>>
>>>>>> Do you or anyone else have another idea ?
>>>>>> Thanks,
>>>>>>
>>>>>>
>>>>>> Loïc
>>>>>>
>>>>>> Loïc CHANEL
>>>>>> Engineering student at TELECOM Nancy
>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>
>>>>>> 2015-07-29 15:34 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>
>>>>>> :
>>>>>>
>>>>>>> No, because I thought the idea of infinite operation was not very
>>>>>>> compatible with the "idle" word (as the operation will not stop running),
>>>>>>> but I'll try :-)
>>>>>>> Thanks for the idea,
>>>>>>>
>>>>>>>
>>>>>>> Loïc
>>>>>>>
>>>>>>> Loïc CHANEL
>>>>>>> Engineering student at TELECOM Nancy
>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>
>>>>>>> 2015-07-29 15:27 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>>>>>>
>>>>>>>> Have you tried hive.server2.idle.operation.timeout?
>>>>>>>>
>>>>>>>> --Xuefu
>>>>>>>>
>>>>>>>> On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel <
>>>>>>>> loic.chanel@telecomnancy.net> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> As I'm trying to build a secured and multi-tenant Hadoop cluster
>>>>>>>>> with Hive, I am desperately trying to set a timeout to Hive requests.
>>>>>>>>> My idea is that some users can make mistakes such as a join with
>>>>>>>>> wrong keys, and therefore start an infinite loop believing that they are
>>>>>>>>> just launching a very heavy job. Therefore, I'd like to set a limit to the
>>>>>>>>> time a request should take, in order to kill the job automatically if it
>>>>>>>>> exceeds it.
>>>>>>>>>
>>>>>>>>> As such a notion cannot be set directly in YARN, I saw that
>>>>>>>>> MapReduce2 provides with its own native timeout property, and I would like
>>>>>>>>> to know if Hive provides with the same property someway.
>>>>>>>>>
>>>>>>>>> Did anyone heard about such a thing ?
>>>>>>>>>
>>>>>>>>> Thanks in advance for your help,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Loïc
>>>>>>>>>
>>>>>>>>> Loïc CHANEL
>>>>>>>>> Engineering student at TELECOM Nancy
>>>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Computation timeout
Posted by Loïc Chanel <lo...@telecomnancy.net>.
My bad, I think I just mixed up the properties.
At the end of the day, everything seems to work as you described.
Thanks a lot !
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-30 9:31 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> Rats, I think I just figured it out.
> #2 Is NEGATIVE 3000, right ? I set it to positive yesterday.
> As for #1, I think it is the default value, so I am not sure I have to set
> it.
>
> Can you confirm that there is a typo on the name of your properties
> (missing last letter) and that is not the actual name of the properties ?
>
> I'll try again and keep you informed
>
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-07-29 20:15 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>
>> this works for me:
>> In hive-site.xml:
>> 1. hive.server2.session.check.interva=3000;
>> 2. hive.server2.idle.operation.timeou=-30000;
>> restart HiveServer2.
>>
>> at beeline, I do "analyze table X compute statistics for columns", which
>> takes longer than 30s. it was aborted by HS2 because of above settings. I
>> guess it didn't work for you because you didn't have #1.
>>
>> --Xuefu
>>
>> On Wed, Jul 29, 2015 at 9:23 AM, Loïc Chanel <
>> loic.chanel@telecomnancy.net> wrote:
>>
>>> I don't think your solution works, as after more than 4 minutes I could
>>> still see logs of my job showing that it was running.
>>> Do you have a way to check that even if the job was running, it was not
>>> being killed by Hive ?
>>> Or another solution ?
>>>
>>> Thanks for your help,
>>>
>>>
>>> Loïc
>>>
>>> Loïc CHANEL
>>> Engineering student at TELECOM Nancy
>>> Trainee at Worldline - Villeurbanne
>>>
>>> 2015-07-29 16:26 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>>>
>>>> Yes, I set it to negative 60.
>>>>
>>>> It's not a problem if the session is killed. That's actually what I try
>>>> to do, because I can't allow to a user to try to end an infinite request.
>>>> Therefore I'll try your solution :)
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> Loïc
>>>>
>>>> Loïc CHANEL
>>>> Engineering student at TELECOM Nancy
>>>> Trainee at Worldline - Villeurbanne
>>>>
>>>> 2015-07-29 16:14 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>>>
>>>>> Okay. To confirm, you set it to negative 60s?
>>>>>
>>>>> The next thing you can try is to set
>>>>> hive.server2.idle.session.timeou=60000 (60sec) and
>>>>> hive.server2.idle.session.check.operation=false. I'm pretty sure this
>>>>> works, but the user's session will be killed though.
>>>>>
>>>>> --Xuefu
>>>>>
>>>>> On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel <
>>>>> loic.chanel@telecomnancy.net> wrote:
>>>>>
>>>>>> I confirm : I just tried hive.server2.idle.operation.timeout setting
>>>>>> it to -60 (seconds), but my veeeeeery slow job have not been killed. The
>>>>>> issue here is "what if another user come and try to submit a MapReduce job
>>>>>> but the cluster is stuck in an infinite loop ?".
>>>>>>
>>>>>> Do you or anyone else have another idea ?
>>>>>> Thanks,
>>>>>>
>>>>>>
>>>>>> Loïc
>>>>>>
>>>>>> Loïc CHANEL
>>>>>> Engineering student at TELECOM Nancy
>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>
>>>>>> 2015-07-29 15:34 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>
>>>>>> :
>>>>>>
>>>>>>> No, because I thought the idea of infinite operation was not very
>>>>>>> compatible with the "idle" word (as the operation will not stop running),
>>>>>>> but I'll try :-)
>>>>>>> Thanks for the idea,
>>>>>>>
>>>>>>>
>>>>>>> Loïc
>>>>>>>
>>>>>>> Loïc CHANEL
>>>>>>> Engineering student at TELECOM Nancy
>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>
>>>>>>> 2015-07-29 15:27 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>>>>>>
>>>>>>>> Have you tried hive.server2.idle.operation.timeout?
>>>>>>>>
>>>>>>>> --Xuefu
>>>>>>>>
>>>>>>>> On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel <
>>>>>>>> loic.chanel@telecomnancy.net> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> As I'm trying to build a secured and multi-tenant Hadoop cluster
>>>>>>>>> with Hive, I am desperately trying to set a timeout to Hive requests.
>>>>>>>>> My idea is that some users can make mistakes such as a join with
>>>>>>>>> wrong keys, and therefore start an infinite loop believing that they are
>>>>>>>>> just launching a very heavy job. Therefore, I'd like to set a limit to the
>>>>>>>>> time a request should take, in order to kill the job automatically if it
>>>>>>>>> exceeds it.
>>>>>>>>>
>>>>>>>>> As such a notion cannot be set directly in YARN, I saw that
>>>>>>>>> MapReduce2 provides with its own native timeout property, and I would like
>>>>>>>>> to know if Hive provides with the same property someway.
>>>>>>>>>
>>>>>>>>> Did anyone heard about such a thing ?
>>>>>>>>>
>>>>>>>>> Thanks in advance for your help,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Loïc
>>>>>>>>>
>>>>>>>>> Loïc CHANEL
>>>>>>>>> Engineering student at TELECOM Nancy
>>>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Computation timeout
Posted by Loïc Chanel <lo...@telecomnancy.net>.
Rats, I think I just figured it out.
#2 Is NEGATIVE 3000, right ? I set it to positive yesterday.
As for #1, I think it is the default value, so I am not sure I have to set
it.
Can you confirm that there is a typo on the name of your properties
(missing last letter) and that is not the actual name of the properties ?
I'll try again and keep you informed
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-29 20:15 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
> this works for me:
> In hive-site.xml:
> 1. hive.server2.session.check.interva=3000;
> 2. hive.server2.idle.operation.timeou=-30000;
> restart HiveServer2.
>
> at beeline, I do "analyze table X compute statistics for columns", which
> takes longer than 30s. it was aborted by HS2 because of above settings. I
> guess it didn't work for you because you didn't have #1.
>
> --Xuefu
>
> On Wed, Jul 29, 2015 at 9:23 AM, Loïc Chanel <loic.chanel@telecomnancy.net
> > wrote:
>
>> I don't think your solution works, as after more than 4 minutes I could
>> still see logs of my job showing that it was running.
>> Do you have a way to check that even if the job was running, it was not
>> being killed by Hive ?
>> Or another solution ?
>>
>> Thanks for your help,
>>
>>
>> Loïc
>>
>> Loïc CHANEL
>> Engineering student at TELECOM Nancy
>> Trainee at Worldline - Villeurbanne
>>
>> 2015-07-29 16:26 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>>
>>> Yes, I set it to negative 60.
>>>
>>> It's not a problem if the session is killed. That's actually what I try
>>> to do, because I can't allow to a user to try to end an infinite request.
>>> Therefore I'll try your solution :)
>>>
>>> Thanks,
>>>
>>>
>>> Loïc
>>>
>>> Loïc CHANEL
>>> Engineering student at TELECOM Nancy
>>> Trainee at Worldline - Villeurbanne
>>>
>>> 2015-07-29 16:14 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>>
>>>> Okay. To confirm, you set it to negative 60s?
>>>>
>>>> The next thing you can try is to set
>>>> hive.server2.idle.session.timeou=60000 (60sec) and
>>>> hive.server2.idle.session.check.operation=false. I'm pretty sure this
>>>> works, but the user's session will be killed though.
>>>>
>>>> --Xuefu
>>>>
>>>> On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel <
>>>> loic.chanel@telecomnancy.net> wrote:
>>>>
>>>>> I confirm : I just tried hive.server2.idle.operation.timeout setting
>>>>> it to -60 (seconds), but my veeeeeery slow job have not been killed. The
>>>>> issue here is "what if another user come and try to submit a MapReduce job
>>>>> but the cluster is stuck in an infinite loop ?".
>>>>>
>>>>> Do you or anyone else have another idea ?
>>>>> Thanks,
>>>>>
>>>>>
>>>>> Loïc
>>>>>
>>>>> Loïc CHANEL
>>>>> Engineering student at TELECOM Nancy
>>>>> Trainee at Worldline - Villeurbanne
>>>>>
>>>>> 2015-07-29 15:34 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>>>>>
>>>>>> No, because I thought the idea of infinite operation was not very
>>>>>> compatible with the "idle" word (as the operation will not stop running),
>>>>>> but I'll try :-)
>>>>>> Thanks for the idea,
>>>>>>
>>>>>>
>>>>>> Loïc
>>>>>>
>>>>>> Loïc CHANEL
>>>>>> Engineering student at TELECOM Nancy
>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>
>>>>>> 2015-07-29 15:27 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>>>>>
>>>>>>> Have you tried hive.server2.idle.operation.timeout?
>>>>>>>
>>>>>>> --Xuefu
>>>>>>>
>>>>>>> On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel <
>>>>>>> loic.chanel@telecomnancy.net> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> As I'm trying to build a secured and multi-tenant Hadoop cluster
>>>>>>>> with Hive, I am desperately trying to set a timeout to Hive requests.
>>>>>>>> My idea is that some users can make mistakes such as a join with
>>>>>>>> wrong keys, and therefore start an infinite loop believing that they are
>>>>>>>> just launching a very heavy job. Therefore, I'd like to set a limit to the
>>>>>>>> time a request should take, in order to kill the job automatically if it
>>>>>>>> exceeds it.
>>>>>>>>
>>>>>>>> As such a notion cannot be set directly in YARN, I saw that
>>>>>>>> MapReduce2 provides with its own native timeout property, and I would like
>>>>>>>> to know if Hive provides with the same property someway.
>>>>>>>>
>>>>>>>> Did anyone heard about such a thing ?
>>>>>>>>
>>>>>>>> Thanks in advance for your help,
>>>>>>>>
>>>>>>>>
>>>>>>>> Loïc
>>>>>>>>
>>>>>>>> Loïc CHANEL
>>>>>>>> Engineering student at TELECOM Nancy
>>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Computation timeout
Posted by Xuefu Zhang <xz...@cloudera.com>.
this works for me:
In hive-site.xml:
1. hive.server2.session.check.interva=3000;
2. hive.server2.idle.operation.timeou=-30000;
restart HiveServer2.
at beeline, I do "analyze table X compute statistics for columns", which
takes longer than 30s. it was aborted by HS2 because of above settings. I
guess it didn't work for you because you didn't have #1.
--Xuefu
On Wed, Jul 29, 2015 at 9:23 AM, Loïc Chanel <lo...@telecomnancy.net>
wrote:
> I don't think your solution works, as after more than 4 minutes I could
> still see logs of my job showing that it was running.
> Do you have a way to check that even if the job was running, it was not
> being killed by Hive ?
> Or another solution ?
>
> Thanks for your help,
>
>
> Loïc
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-07-29 16:26 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>
>> Yes, I set it to negative 60.
>>
>> It's not a problem if the session is killed. That's actually what I try
>> to do, because I can't allow to a user to try to end an infinite request.
>> Therefore I'll try your solution :)
>>
>> Thanks,
>>
>>
>> Loïc
>>
>> Loïc CHANEL
>> Engineering student at TELECOM Nancy
>> Trainee at Worldline - Villeurbanne
>>
>> 2015-07-29 16:14 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>
>>> Okay. To confirm, you set it to negative 60s?
>>>
>>> The next thing you can try is to set
>>> hive.server2.idle.session.timeou=60000 (60sec) and
>>> hive.server2.idle.session.check.operation=false. I'm pretty sure this
>>> works, but the user's session will be killed though.
>>>
>>> --Xuefu
>>>
>>> On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel <
>>> loic.chanel@telecomnancy.net> wrote:
>>>
>>>> I confirm : I just tried hive.server2.idle.operation.timeout setting it
>>>> to -60 (seconds), but my veeeeeery slow job have not been killed. The issue
>>>> here is "what if another user come and try to submit a MapReduce job but
>>>> the cluster is stuck in an infinite loop ?".
>>>>
>>>> Do you or anyone else have another idea ?
>>>> Thanks,
>>>>
>>>>
>>>> Loïc
>>>>
>>>> Loïc CHANEL
>>>> Engineering student at TELECOM Nancy
>>>> Trainee at Worldline - Villeurbanne
>>>>
>>>> 2015-07-29 15:34 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>>>>
>>>>> No, because I thought the idea of infinite operation was not very
>>>>> compatible with the "idle" word (as the operation will not stop running),
>>>>> but I'll try :-)
>>>>> Thanks for the idea,
>>>>>
>>>>>
>>>>> Loïc
>>>>>
>>>>> Loïc CHANEL
>>>>> Engineering student at TELECOM Nancy
>>>>> Trainee at Worldline - Villeurbanne
>>>>>
>>>>> 2015-07-29 15:27 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>>>>
>>>>>> Have you tried hive.server2.idle.operation.timeout?
>>>>>>
>>>>>> --Xuefu
>>>>>>
>>>>>> On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel <
>>>>>> loic.chanel@telecomnancy.net> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> As I'm trying to build a secured and multi-tenant Hadoop cluster
>>>>>>> with Hive, I am desperately trying to set a timeout to Hive requests.
>>>>>>> My idea is that some users can make mistakes such as a join with
>>>>>>> wrong keys, and therefore start an infinite loop believing that they are
>>>>>>> just launching a very heavy job. Therefore, I'd like to set a limit to the
>>>>>>> time a request should take, in order to kill the job automatically if it
>>>>>>> exceeds it.
>>>>>>>
>>>>>>> As such a notion cannot be set directly in YARN, I saw that
>>>>>>> MapReduce2 provides with its own native timeout property, and I would like
>>>>>>> to know if Hive provides with the same property someway.
>>>>>>>
>>>>>>> Did anyone heard about such a thing ?
>>>>>>>
>>>>>>> Thanks in advance for your help,
>>>>>>>
>>>>>>>
>>>>>>> Loïc
>>>>>>>
>>>>>>> Loïc CHANEL
>>>>>>> Engineering student at TELECOM Nancy
>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Computation timeout
Posted by Loïc Chanel <lo...@telecomnancy.net>.
I don't think your solution works, as after more than 4 minutes I could
still see logs of my job showing that it was running.
Do you have a way to check that even if the job was running, it was not
being killed by Hive ?
Or another solution ?
Thanks for your help,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-29 16:26 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> Yes, I set it to negative 60.
>
> It's not a problem if the session is killed. That's actually what I try to
> do, because I can't allow to a user to try to end an infinite request.
> Therefore I'll try your solution :)
>
> Thanks,
>
>
> Loïc
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-07-29 16:14 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>
>> Okay. To confirm, you set it to negative 60s?
>>
>> The next thing you can try is to set
>> hive.server2.idle.session.timeou=60000 (60sec) and
>> hive.server2.idle.session.check.operation=false. I'm pretty sure this
>> works, but the user's session will be killed though.
>>
>> --Xuefu
>>
>> On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel <
>> loic.chanel@telecomnancy.net> wrote:
>>
>>> I confirm : I just tried hive.server2.idle.operation.timeout setting it
>>> to -60 (seconds), but my veeeeeery slow job have not been killed. The issue
>>> here is "what if another user come and try to submit a MapReduce job but
>>> the cluster is stuck in an infinite loop ?".
>>>
>>> Do you or anyone else have another idea ?
>>> Thanks,
>>>
>>>
>>> Loïc
>>>
>>> Loïc CHANEL
>>> Engineering student at TELECOM Nancy
>>> Trainee at Worldline - Villeurbanne
>>>
>>> 2015-07-29 15:34 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>>>
>>>> No, because I thought the idea of infinite operation was not very
>>>> compatible with the "idle" word (as the operation will not stop running),
>>>> but I'll try :-)
>>>> Thanks for the idea,
>>>>
>>>>
>>>> Loïc
>>>>
>>>> Loïc CHANEL
>>>> Engineering student at TELECOM Nancy
>>>> Trainee at Worldline - Villeurbanne
>>>>
>>>> 2015-07-29 15:27 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>>>
>>>>> Have you tried hive.server2.idle.operation.timeout?
>>>>>
>>>>> --Xuefu
>>>>>
>>>>> On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel <
>>>>> loic.chanel@telecomnancy.net> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> As I'm trying to build a secured and multi-tenant Hadoop cluster with
>>>>>> Hive, I am desperately trying to set a timeout to Hive requests.
>>>>>> My idea is that some users can make mistakes such as a join with
>>>>>> wrong keys, and therefore start an infinite loop believing that they are
>>>>>> just launching a very heavy job. Therefore, I'd like to set a limit to the
>>>>>> time a request should take, in order to kill the job automatically if it
>>>>>> exceeds it.
>>>>>>
>>>>>> As such a notion cannot be set directly in YARN, I saw that
>>>>>> MapReduce2 provides with its own native timeout property, and I would like
>>>>>> to know if Hive provides with the same property someway.
>>>>>>
>>>>>> Did anyone heard about such a thing ?
>>>>>>
>>>>>> Thanks in advance for your help,
>>>>>>
>>>>>>
>>>>>> Loïc
>>>>>>
>>>>>> Loïc CHANEL
>>>>>> Engineering student at TELECOM Nancy
>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Computation timeout
Posted by Loïc Chanel <lo...@telecomnancy.net>.
Yes, I set it to negative 60.
It's not a problem if the session is killed. That's actually what I try to
do, because I can't allow to a user to try to end an infinite request.
Therefore I'll try your solution :)
Thanks,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-29 16:14 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
> Okay. To confirm, you set it to negative 60s?
>
> The next thing you can try is to set
> hive.server2.idle.session.timeou=60000 (60sec) and
> hive.server2.idle.session.check.operation=false. I'm pretty sure this
> works, but the user's session will be killed though.
>
> --Xuefu
>
> On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel <loic.chanel@telecomnancy.net
> > wrote:
>
>> I confirm : I just tried hive.server2.idle.operation.timeout setting it
>> to -60 (seconds), but my veeeeeery slow job have not been killed. The issue
>> here is "what if another user come and try to submit a MapReduce job but
>> the cluster is stuck in an infinite loop ?".
>>
>> Do you or anyone else have another idea ?
>> Thanks,
>>
>>
>> Loïc
>>
>> Loïc CHANEL
>> Engineering student at TELECOM Nancy
>> Trainee at Worldline - Villeurbanne
>>
>> 2015-07-29 15:34 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>>
>>> No, because I thought the idea of infinite operation was not very
>>> compatible with the "idle" word (as the operation will not stop running),
>>> but I'll try :-)
>>> Thanks for the idea,
>>>
>>>
>>> Loïc
>>>
>>> Loïc CHANEL
>>> Engineering student at TELECOM Nancy
>>> Trainee at Worldline - Villeurbanne
>>>
>>> 2015-07-29 15:27 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>>
>>>> Have you tried hive.server2.idle.operation.timeout?
>>>>
>>>> --Xuefu
>>>>
>>>> On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel <
>>>> loic.chanel@telecomnancy.net> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> As I'm trying to build a secured and multi-tenant Hadoop cluster with
>>>>> Hive, I am desperately trying to set a timeout to Hive requests.
>>>>> My idea is that some users can make mistakes such as a join with wrong
>>>>> keys, and therefore start an infinite loop believing that they are just
>>>>> launching a very heavy job. Therefore, I'd like to set a limit to the time
>>>>> a request should take, in order to kill the job automatically if it exceeds
>>>>> it.
>>>>>
>>>>> As such a notion cannot be set directly in YARN, I saw that MapReduce2
>>>>> provides with its own native timeout property, and I would like to know if
>>>>> Hive provides with the same property someway.
>>>>>
>>>>> Did anyone heard about such a thing ?
>>>>>
>>>>> Thanks in advance for your help,
>>>>>
>>>>>
>>>>> Loïc
>>>>>
>>>>> Loïc CHANEL
>>>>> Engineering student at TELECOM Nancy
>>>>> Trainee at Worldline - Villeurbanne
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Computation timeout
Posted by Xuefu Zhang <xz...@cloudera.com>.
Okay. To confirm, you set it to negative 60s?
The next thing you can try is to set hive.server2.idle.session.timeou=60000
(60sec) and hive.server2.idle.session.check.operation=false. I'm pretty
sure this works, but the user's session will be killed though.
--Xuefu
On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel <lo...@telecomnancy.net>
wrote:
> I confirm : I just tried hive.server2.idle.operation.timeout setting it to
> -60 (seconds), but my veeeeeery slow job have not been killed. The issue
> here is "what if another user come and try to submit a MapReduce job but
> the cluster is stuck in an infinite loop ?".
>
> Do you or anyone else have another idea ?
> Thanks,
>
>
> Loïc
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-07-29 15:34 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>
>> No, because I thought the idea of infinite operation was not very
>> compatible with the "idle" word (as the operation will not stop running),
>> but I'll try :-)
>> Thanks for the idea,
>>
>>
>> Loïc
>>
>> Loïc CHANEL
>> Engineering student at TELECOM Nancy
>> Trainee at Worldline - Villeurbanne
>>
>> 2015-07-29 15:27 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>>
>>> Have you tried hive.server2.idle.operation.timeout?
>>>
>>> --Xuefu
>>>
>>> On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel <
>>> loic.chanel@telecomnancy.net> wrote:
>>>
>>>> Hi all,
>>>>
>>>> As I'm trying to build a secured and multi-tenant Hadoop cluster with
>>>> Hive, I am desperately trying to set a timeout to Hive requests.
>>>> My idea is that some users can make mistakes such as a join with wrong
>>>> keys, and therefore start an infinite loop believing that they are just
>>>> launching a very heavy job. Therefore, I'd like to set a limit to the time
>>>> a request should take, in order to kill the job automatically if it exceeds
>>>> it.
>>>>
>>>> As such a notion cannot be set directly in YARN, I saw that MapReduce2
>>>> provides with its own native timeout property, and I would like to know if
>>>> Hive provides with the same property someway.
>>>>
>>>> Did anyone heard about such a thing ?
>>>>
>>>> Thanks in advance for your help,
>>>>
>>>>
>>>> Loïc
>>>>
>>>> Loïc CHANEL
>>>> Engineering student at TELECOM Nancy
>>>> Trainee at Worldline - Villeurbanne
>>>>
>>>
>>>
>>
>
Re: Computation timeout
Posted by Loïc Chanel <lo...@telecomnancy.net>.
I confirm : I just tried hive.server2.idle.operation.timeout setting it to
-60 (seconds), but my veeeeeery slow job have not been killed. The issue
here is "what if another user come and try to submit a MapReduce job but
the cluster is stuck in an infinite loop ?".
Do you or anyone else have another idea ?
Thanks,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-29 15:34 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> No, because I thought the idea of infinite operation was not very
> compatible with the "idle" word (as the operation will not stop running),
> but I'll try :-)
> Thanks for the idea,
>
>
> Loïc
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-07-29 15:27 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
>
>> Have you tried hive.server2.idle.operation.timeout?
>>
>> --Xuefu
>>
>> On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel <
>> loic.chanel@telecomnancy.net> wrote:
>>
>>> Hi all,
>>>
>>> As I'm trying to build a secured and multi-tenant Hadoop cluster with
>>> Hive, I am desperately trying to set a timeout to Hive requests.
>>> My idea is that some users can make mistakes such as a join with wrong
>>> keys, and therefore start an infinite loop believing that they are just
>>> launching a very heavy job. Therefore, I'd like to set a limit to the time
>>> a request should take, in order to kill the job automatically if it exceeds
>>> it.
>>>
>>> As such a notion cannot be set directly in YARN, I saw that MapReduce2
>>> provides with its own native timeout property, and I would like to know if
>>> Hive provides with the same property someway.
>>>
>>> Did anyone heard about such a thing ?
>>>
>>> Thanks in advance for your help,
>>>
>>>
>>> Loïc
>>>
>>> Loïc CHANEL
>>> Engineering student at TELECOM Nancy
>>> Trainee at Worldline - Villeurbanne
>>>
>>
>>
>
Re: Computation timeout
Posted by Loïc Chanel <lo...@telecomnancy.net>.
No, because I thought the idea of infinite operation was not very
compatible with the "idle" word (as the operation will not stop running),
but I'll try :-)
Thanks for the idea,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-07-29 15:27 GMT+02:00 Xuefu Zhang <xz...@cloudera.com>:
> Have you tried hive.server2.idle.operation.timeout?
>
> --Xuefu
>
> On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel <loic.chanel@telecomnancy.net
> > wrote:
>
>> Hi all,
>>
>> As I'm trying to build a secured and multi-tenant Hadoop cluster with
>> Hive, I am desperately trying to set a timeout to Hive requests.
>> My idea is that some users can make mistakes such as a join with wrong
>> keys, and therefore start an infinite loop believing that they are just
>> launching a very heavy job. Therefore, I'd like to set a limit to the time
>> a request should take, in order to kill the job automatically if it exceeds
>> it.
>>
>> As such a notion cannot be set directly in YARN, I saw that MapReduce2
>> provides with its own native timeout property, and I would like to know if
>> Hive provides with the same property someway.
>>
>> Did anyone heard about such a thing ?
>>
>> Thanks in advance for your help,
>>
>>
>> Loïc
>>
>> Loïc CHANEL
>> Engineering student at TELECOM Nancy
>> Trainee at Worldline - Villeurbanne
>>
>
>
Re: Computation timeout
Posted by Xuefu Zhang <xz...@cloudera.com>.
Have you tried hive.server2.idle.operation.timeout?
--Xuefu
On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel <lo...@telecomnancy.net>
wrote:
> Hi all,
>
> As I'm trying to build a secured and multi-tenant Hadoop cluster with
> Hive, I am desperately trying to set a timeout to Hive requests.
> My idea is that some users can make mistakes such as a join with wrong
> keys, and therefore start an infinite loop believing that they are just
> launching a very heavy job. Therefore, I'd like to set a limit to the time
> a request should take, in order to kill the job automatically if it exceeds
> it.
>
> As such a notion cannot be set directly in YARN, I saw that MapReduce2
> provides with its own native timeout property, and I would like to know if
> Hive provides with the same property someway.
>
> Did anyone heard about such a thing ?
>
> Thanks in advance for your help,
>
>
> Loïc
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>