You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Abhilash L L <ab...@capillarytech.com> on 2014/09/06 08:37:22 UTC
Hive Mapred local task distribution
Hello,
We are using Hive 0.11 connecting to it via Hive Thrift server 2.
A lot of our queries are launching map red local tasks, which is good
and expected. Since we are firing queries in parallel, these tasks are all
starting on same host and consuming a lot of resources.
Is there a way to distribute these on different nodes ?
Or the only possibility is to do it in app layer and load balance it on
few of the thrift servers ?
Please do let me know in case I have to share any more information
regarding the setup etc.
Regards,
Abhilash L L
Capillary Technologies
M:919886208262
abhilash@capillarytech.com | www.capillarytech.com
Email from people at capillarytech.com may not represent official policy of
Capillary Technologies unless explicitly stated. Please see our
Corporate-Email-Policy
<http://support.capillary.co.in/policy-public/Corporate-Email-Policy.pdf>
for details. Contents of this email are confidential. Please contact the
Sender if you have received this email in error.
--
Email from people at capillarytech.com may not represent official policy of
Capillary Technologies unless explicitly stated. Please see our
Corporate-Email-Policy for details.Contents of this email are confidential.
Please contact the Sender if you have received this email in error.
Re: Hive Mapred local task distribution
Posted by Xuefu Zhang <xz...@cloudera.com>.
You might be able to control what tasks to run locally. However, once they
run locally, they have to do so on HiveServer2 host.
It's possible to run the local tasks in separate JVMs. Still, the same host.
--Xuefu
On Sat, Sep 6, 2014 at 7:51 AM, Abhilash L L <ab...@capillarytech.com>
wrote:
> Hi Xuefu,
>
> Yea, currently we have only one HiveServer2 host where the map red
> local tasks run.
>
> Any other solution other than LBing it ?
>
>
> Regards,
> Abhilash L L
> Capillary Technologies
> M:919886208262
> abhilash@capillarytech.com | www.capillarytech.com
>
> Email from people at capillarytech.com may not represent official policy
> of Capillary Technologies unless explicitly stated. Please see our
> Corporate-Email-Policy
> <http://support.capillary.co.in/policy-public/Corporate-Email-Policy.pdf>
> for details. Contents of this email are confidential. Please contact the
> Sender if you have received this email in error.
>
>
>
> On Sat, Sep 6, 2014 at 7:53 PM, Xuefu Zhang <xz...@cloudera.com> wrote:
>
>> By "same host", don't you mean your HiveServer2 host? One solution is to
>> have multiple HiveServer2 instances and do load balance among them.
>>
>> --Xuefu
>>
>>
>> On Fri, Sep 5, 2014 at 11:37 PM, Abhilash L L <abhilash@capillarytech.com
>> > wrote:
>>
>>> Hello,
>>>
>>> We are using Hive 0.11 connecting to it via Hive Thrift server 2.
>>>
>>> A lot of our queries are launching map red local tasks, which is good
>>> and expected. Since we are firing queries in parallel, these tasks are all
>>> starting on same host and consuming a lot of resources.
>>>
>>> Is there a way to distribute these on different nodes ?
>>>
>>> Or the only possibility is to do it in app layer and load balance it
>>> on few of the thrift servers ?
>>>
>>> Please do let me know in case I have to share any more information
>>> regarding the setup etc.
>>>
>>> Regards,
>>> Abhilash L L
>>> Capillary Technologies
>>> M:919886208262
>>> abhilash@capillarytech.com | www.capillarytech.com
>>>
>>> Email from people at capillarytech.com may not represent official
>>> policy of Capillary Technologies unless explicitly stated. Please see our
>>> Corporate-Email-Policy
>>> <http://support.capillary.co.in/policy-public/Corporate-Email-Policy.pdf>
>>> for details. Contents of this email are confidential. Please contact the
>>> Sender if you have received this email in error.
>>>
>>>
>>> Email from people at capillarytech.com may not represent official
>>> policy of Capillary Technologies unless explicitly stated. Please see our
>>> Corporate-Email-Policy for details.Contents of this email are confidential.
>>> Please contact the Sender if you have received this email in error.
>>>
>>
>>
>
> Email from people at capillarytech.com may not represent official policy
> of Capillary Technologies unless explicitly stated. Please see our
> Corporate-Email-Policy for details.Contents of this email are confidential.
> Please contact the Sender if you have received this email in error.
>
Re: Hive Mapred local task distribution
Posted by Abhilash L L <ab...@capillarytech.com>.
Hi Xuefu,
Yea, currently we have only one HiveServer2 host where the map red
local tasks run.
Any other solution other than LBing it ?
Regards,
Abhilash L L
Capillary Technologies
M:919886208262
abhilash@capillarytech.com | www.capillarytech.com
Email from people at capillarytech.com may not represent official policy of
Capillary Technologies unless explicitly stated. Please see our
Corporate-Email-Policy
<http://support.capillary.co.in/policy-public/Corporate-Email-Policy.pdf>
for details. Contents of this email are confidential. Please contact the
Sender if you have received this email in error.
On Sat, Sep 6, 2014 at 7:53 PM, Xuefu Zhang <xz...@cloudera.com> wrote:
> By "same host", don't you mean your HiveServer2 host? One solution is to
> have multiple HiveServer2 instances and do load balance among them.
>
> --Xuefu
>
>
> On Fri, Sep 5, 2014 at 11:37 PM, Abhilash L L <ab...@capillarytech.com>
> wrote:
>
>> Hello,
>>
>> We are using Hive 0.11 connecting to it via Hive Thrift server 2.
>>
>> A lot of our queries are launching map red local tasks, which is good
>> and expected. Since we are firing queries in parallel, these tasks are all
>> starting on same host and consuming a lot of resources.
>>
>> Is there a way to distribute these on different nodes ?
>>
>> Or the only possibility is to do it in app layer and load balance it
>> on few of the thrift servers ?
>>
>> Please do let me know in case I have to share any more information
>> regarding the setup etc.
>>
>> Regards,
>> Abhilash L L
>> Capillary Technologies
>> M:919886208262
>> abhilash@capillarytech.com | www.capillarytech.com
>>
>> Email from people at capillarytech.com may not represent official policy
>> of Capillary Technologies unless explicitly stated. Please see our
>> Corporate-Email-Policy
>> <http://support.capillary.co.in/policy-public/Corporate-Email-Policy.pdf>
>> for details. Contents of this email are confidential. Please contact the
>> Sender if you have received this email in error.
>>
>>
>> Email from people at capillarytech.com may not represent official policy
>> of Capillary Technologies unless explicitly stated. Please see our
>> Corporate-Email-Policy for details.Contents of this email are confidential.
>> Please contact the Sender if you have received this email in error.
>>
>
>
--
Email from people at capillarytech.com may not represent official policy of
Capillary Technologies unless explicitly stated. Please see our
Corporate-Email-Policy for details.Contents of this email are confidential.
Please contact the Sender if you have received this email in error.
Re: Hive Mapred local task distribution
Posted by Xuefu Zhang <xz...@cloudera.com>.
By "same host", don't you mean your HiveServer2 host? One solution is to
have multiple HiveServer2 instances and do load balance among them.
--Xuefu
On Fri, Sep 5, 2014 at 11:37 PM, Abhilash L L <ab...@capillarytech.com>
wrote:
> Hello,
>
> We are using Hive 0.11 connecting to it via Hive Thrift server 2.
>
> A lot of our queries are launching map red local tasks, which is good
> and expected. Since we are firing queries in parallel, these tasks are all
> starting on same host and consuming a lot of resources.
>
> Is there a way to distribute these on different nodes ?
>
> Or the only possibility is to do it in app layer and load balance it
> on few of the thrift servers ?
>
> Please do let me know in case I have to share any more information
> regarding the setup etc.
>
> Regards,
> Abhilash L L
> Capillary Technologies
> M:919886208262
> abhilash@capillarytech.com | www.capillarytech.com
>
> Email from people at capillarytech.com may not represent official policy
> of Capillary Technologies unless explicitly stated. Please see our
> Corporate-Email-Policy
> <http://support.capillary.co.in/policy-public/Corporate-Email-Policy.pdf>
> for details. Contents of this email are confidential. Please contact the
> Sender if you have received this email in error.
>
>
> Email from people at capillarytech.com may not represent official policy
> of Capillary Technologies unless explicitly stated. Please see our
> Corporate-Email-Policy for details.Contents of this email are confidential.
> Please contact the Sender if you have received this email in error.
>