You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Margus Roo <ma...@roo.ee> on 2016/02/26 15:40:26 UTC
Creating hive external table gives GC pool 'PS MarkSweep' had
collection(s)
Hi
I try to create external table and in the location there are 8960 small
files
And I am getting every time something like that:
GC pool 'PS MarkSweep' had collection(s): count=1 time=1672ms
GC pool 'PS Scavenge' had collection(s): count=1 time=45ms
2016-02-26 15:18:29,721 INFO
[org.apache.hadoop.util.JvmPauseMonitor$Monitor@3af36922]:
util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in
JVM or host machine (eg GC): pause of approximately 1995ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=1936ms
And all 16 cpu cores are in 100% and all 16G memory almost gone.
Helps only if I restart hive server.
I use Hive 1.2.1 from HDP-2.3 by packaged by Hortonworks
hive.tez.java.opts=-server -Djava.net.preferIPv4Stack=true
-XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB
-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps
--
Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 48 780
Re: Creating hive external table gives GC pool 'PS MarkSweep' had
collection(s)
Posted by Margus Roo <ma...@roo.ee>.
Hi
Found that in my config there was
hive.exec.dynamic.partition = true;
I turned it false and most of the times I can create table now but not
every time.
Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 48 780
On 29/02/16 09:27, Margus Roo wrote:
> Hi
>
> Can someone confirm that Hive checks files in destination directory
> before creating external table?
> At the moment in Hive 1.2.1 end user can just easily kill whole Hive
> server creating external table and pointing to directory where are
> loads of files.
>
> Margus (margusja) Roo
> http://margus.roo.ee
> skype: margusja
> +372 51 48 780
>
> On 26/02/16 17:27, Margus Roo wrote:
>> Basically the question is:
>> Does Hive checks files in location before creating table?
>> Because if I move files away before creating tables it works and
>> after table is created I can move files back and all works :)
>>
>> Margus (margusja) Roo
>> http://margus.roo.ee
>> skype: margusja
>> +372 51 48 780
>>
>> On 26/02/16 16:40, Margus Roo wrote:
>>> Hi
>>>
>>> I try to create external table and in the location there are 8960
>>> small files
>>>
>>> And I am getting every time something like that:
>>> GC pool 'PS MarkSweep' had collection(s): count=1 time=1672ms
>>> GC pool 'PS Scavenge' had collection(s): count=1 time=45ms
>>> 2016-02-26 15:18:29,721 INFO
>>> [org.apache.hadoop.util.JvmPauseMonitor$Monitor@3af36922]:
>>> util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected
>>> pause in JVM or host machine (eg GC): pause of approximately 1995ms
>>> GC pool 'PS MarkSweep' had collection(s): count=1 time=1936ms
>>>
>>> And all 16 cpu cores are in 100% and all 16G memory almost gone.
>>>
>>> Helps only if I restart hive server.
>>>
>>> I use Hive 1.2.1 from HDP-2.3 by packaged by Hortonworks
>>>
>>> hive.tez.java.opts=-server -Djava.net.preferIPv4Stack=true
>>> -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB
>>> -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps
>>>
>>>
>>
>
Re: Creating hive external table gives GC pool 'PS MarkSweep' had
collection(s)
Posted by Margus Roo <ma...@roo.ee>.
Hi
Can someone confirm that Hive checks files in destination directory
before creating external table?
At the moment in Hive 1.2.1 end user can just easily kill whole Hive
server creating external table and pointing to directory where are loads
of files.
Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 48 780
On 26/02/16 17:27, Margus Roo wrote:
> Basically the question is:
> Does Hive checks files in location before creating table?
> Because if I move files away before creating tables it works and after
> table is created I can move files back and all works :)
>
> Margus (margusja) Roo
> http://margus.roo.ee
> skype: margusja
> +372 51 48 780
>
> On 26/02/16 16:40, Margus Roo wrote:
>> Hi
>>
>> I try to create external table and in the location there are 8960
>> small files
>>
>> And I am getting every time something like that:
>> GC pool 'PS MarkSweep' had collection(s): count=1 time=1672ms
>> GC pool 'PS Scavenge' had collection(s): count=1 time=45ms
>> 2016-02-26 15:18:29,721 INFO
>> [org.apache.hadoop.util.JvmPauseMonitor$Monitor@3af36922]:
>> util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause
>> in JVM or host machine (eg GC): pause of approximately 1995ms
>> GC pool 'PS MarkSweep' had collection(s): count=1 time=1936ms
>>
>> And all 16 cpu cores are in 100% and all 16G memory almost gone.
>>
>> Helps only if I restart hive server.
>>
>> I use Hive 1.2.1 from HDP-2.3 by packaged by Hortonworks
>>
>> hive.tez.java.opts=-server -Djava.net.preferIPv4Stack=true
>> -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB
>> -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps
>>
>>
>
Re: Creating hive external table gives GC pool 'PS MarkSweep' had
collection(s)
Posted by Margus Roo <ma...@roo.ee>.
Basically the question is:
Does Hive checks files in location before creating table?
Because if I move files away before creating tables it works and after
table is created I can move files back and all works :)
Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 48 780
On 26/02/16 16:40, Margus Roo wrote:
> Hi
>
> I try to create external table and in the location there are 8960
> small files
>
> And I am getting every time something like that:
> GC pool 'PS MarkSweep' had collection(s): count=1 time=1672ms
> GC pool 'PS Scavenge' had collection(s): count=1 time=45ms
> 2016-02-26 15:18:29,721 INFO
> [org.apache.hadoop.util.JvmPauseMonitor$Monitor@3af36922]:
> util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause
> in JVM or host machine (eg GC): pause of approximately 1995ms
> GC pool 'PS MarkSweep' had collection(s): count=1 time=1936ms
>
> And all 16 cpu cores are in 100% and all 16G memory almost gone.
>
> Helps only if I restart hive server.
>
> I use Hive 1.2.1 from HDP-2.3 by packaged by Hortonworks
>
> hive.tez.java.opts=-server -Djava.net.preferIPv4Stack=true
> -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB
> -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps
>
>