You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Jone Zhang <jo...@gmail.com> on 2015/10/23 07:32:05 UTC
Hive on Spark
1.How can i set Storage Level when i use Hive on Spark?
2.Do Spark have any intention of dynamically determined Hive on MapReduce
or Hive on Spark, base on SQL features.
Thanks in advance
Best regards
Re: Hive on Spark
Posted by Xuefu Zhang <xz...@cloudera.com>.
you need to increase spark.yarn.executor.memoryOverhead. it has nothing to
do with storage layer.
--Xuefu
On Fri, Oct 23, 2015 at 4:49 AM, Jone Zhang <jo...@gmail.com> wrote:
> I get an the error every time while I run a query on a large data set. I
> think use MEMORY_AND_DISK can avoid this problem under the limited
> resources.
> "15/10/23 17:37:13 Reporter WARN
> org.apache.spark.deploy.yarn.YarnAllocator>> Container killed by YARN for
> exceeding memory limits. 7.6 GB of 7.5 GB physical memory used. Consider
> boosting spark.yarn.executor.memoryOverhead."
>
> 2015-10-23 19:40 GMT+08:00 Xuefu Zhang <xz...@cloudera.com>:
>
>> Yeah. for that, you cannot really cache anything through Hive on Spark.
>> Could you detail more what you want to achieve?
>>
>> When needed, Hive on Spark uses memory+disk for storage level.
>>
>> On Fri, Oct 23, 2015 at 4:29 AM, Jone Zhang <jo...@gmail.com>
>> wrote:
>>
>>> 1.But It's no way to set Storage Level through properties file in
>>> spark, Spark provided "def persist(newLevel: StorageLevel)"
>>> api only...
>>>
>>> 2015-10-23 19:03 GMT+08:00 Xuefu Zhang <xz...@cloudera.com>:
>>>
>>>> quick answers:
>>>> 1. you can pretty much set any spark configuration at hive using set
>>>> command.
>>>> 2. no. you have to make the call.
>>>>
>>>>
>>>>
>>>> On Thu, Oct 22, 2015 at 10:32 PM, Jone Zhang <jo...@gmail.com>
>>>> wrote:
>>>>
>>>>> 1.How can i set Storage Level when i use Hive on Spark?
>>>>> 2.Do Spark have any intention of dynamically determined Hive on
>>>>> MapReduce or Hive on Spark, base on SQL features.
>>>>>
>>>>> Thanks in advance
>>>>> Best regards
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Hive on Spark
Posted by Jone Zhang <jo...@gmail.com>.
I get an the error every time while I run a query on a large data set. I
think use MEMORY_AND_DISK can avoid this problem under the limited
resources.
"15/10/23 17:37:13 Reporter WARN
org.apache.spark.deploy.yarn.YarnAllocator>> Container killed by YARN for
exceeding memory limits. 7.6 GB of 7.5 GB physical memory used. Consider
boosting spark.yarn.executor.memoryOverhead."
2015-10-23 19:40 GMT+08:00 Xuefu Zhang <xz...@cloudera.com>:
> Yeah. for that, you cannot really cache anything through Hive on Spark.
> Could you detail more what you want to achieve?
>
> When needed, Hive on Spark uses memory+disk for storage level.
>
> On Fri, Oct 23, 2015 at 4:29 AM, Jone Zhang <jo...@gmail.com>
> wrote:
>
>> 1.But It's no way to set Storage Level through properties file in spark,
>> Spark provided "def persist(newLevel: StorageLevel)"
>> api only...
>>
>> 2015-10-23 19:03 GMT+08:00 Xuefu Zhang <xz...@cloudera.com>:
>>
>>> quick answers:
>>> 1. you can pretty much set any spark configuration at hive using set
>>> command.
>>> 2. no. you have to make the call.
>>>
>>>
>>>
>>> On Thu, Oct 22, 2015 at 10:32 PM, Jone Zhang <jo...@gmail.com>
>>> wrote:
>>>
>>>> 1.How can i set Storage Level when i use Hive on Spark?
>>>> 2.Do Spark have any intention of dynamically determined Hive on
>>>> MapReduce or Hive on Spark, base on SQL features.
>>>>
>>>> Thanks in advance
>>>> Best regards
>>>>
>>>
>>>
>>
>
Re: Hive on Spark
Posted by Xuefu Zhang <xz...@cloudera.com>.
Yeah. for that, you cannot really cache anything through Hive on Spark.
Could you detail more what you want to achieve?
When needed, Hive on Spark uses memory+disk for storage level.
On Fri, Oct 23, 2015 at 4:29 AM, Jone Zhang <jo...@gmail.com> wrote:
> 1.But It's no way to set Storage Level through properties file in spark,
> Spark provided "def persist(newLevel: StorageLevel)"
> api only...
>
> 2015-10-23 19:03 GMT+08:00 Xuefu Zhang <xz...@cloudera.com>:
>
>> quick answers:
>> 1. you can pretty much set any spark configuration at hive using set
>> command.
>> 2. no. you have to make the call.
>>
>>
>>
>> On Thu, Oct 22, 2015 at 10:32 PM, Jone Zhang <jo...@gmail.com>
>> wrote:
>>
>>> 1.How can i set Storage Level when i use Hive on Spark?
>>> 2.Do Spark have any intention of dynamically determined Hive on
>>> MapReduce or Hive on Spark, base on SQL features.
>>>
>>> Thanks in advance
>>> Best regards
>>>
>>
>>
>
Re: Hive on Spark
Posted by Jone Zhang <jo...@gmail.com>.
1.But It's no way to set Storage Level through properties file in spark,
Spark provided "def persist(newLevel: StorageLevel)"
api only...
2015-10-23 19:03 GMT+08:00 Xuefu Zhang <xz...@cloudera.com>:
> quick answers:
> 1. you can pretty much set any spark configuration at hive using set
> command.
> 2. no. you have to make the call.
>
>
>
> On Thu, Oct 22, 2015 at 10:32 PM, Jone Zhang <jo...@gmail.com>
> wrote:
>
>> 1.How can i set Storage Level when i use Hive on Spark?
>> 2.Do Spark have any intention of dynamically determined Hive on
>> MapReduce or Hive on Spark, base on SQL features.
>>
>> Thanks in advance
>> Best regards
>>
>
>
Re: Hive on Spark
Posted by Xuefu Zhang <xz...@cloudera.com>.
quick answers:
1. you can pretty much set any spark configuration at hive using set
command.
2. no. you have to make the call.
On Thu, Oct 22, 2015 at 10:32 PM, Jone Zhang <jo...@gmail.com> wrote:
> 1.How can i set Storage Level when i use Hive on Spark?
> 2.Do Spark have any intention of dynamically determined Hive on MapReduce
> or Hive on Spark, base on SQL features.
>
> Thanks in advance
> Best regards
>