You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Punit Shetty <pu...@gmail.com> on 2017/03/08 23:47:40 UTC
Missing column headers from S3 file
Hi,
I'm trying to get the column headers into a S3 file.
When I run this on the hive command line:
set hive.cli.print.header=true;
select * from abc;
Things work fine, I'm able to see the column headers with the table data on
the console.
But my query is, when I try something like this:
set hive.cli.print.header=true;
CREATE TABLE test123 ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES
TERMINATED BY '\n' STORED AS TEXTFILE LOCATION 's3://somelocation' AS
select * abc;
The column headers are missing from the S3 file.
Is this the expected behaviour or am I missing something?
Thanks,
Punit
Re: Why direct memory should be smaller than llap size
Posted by Prasanth Jayachandran <pj...@hortonworks.com>.
The “size” here means the container size. You are requesting for a 12GB container but max container size in your cluster is configured to 8GB.
Xmx + cache should be less than size which should be less than max container size.
Also, set the —executors/—iothreads to a value <=max vcores.
—size 8g —Xmx 4g —cache 3200m — executors 4 —iothreads 4
This will request for 8gb container out of which 4gb is shared by 4 executor threads and 3200 mb is used for off heap cache.
Thanks
Prasanth
On Mar 14, 2017, at 1:01 AM, 邓志华 <zh...@CREDITEASE.CN>> wrote:
Thanks @pjayachandran, I try to add xmx to specify the executors memory.
hive --service llap --name llap_service --instances 16 --size 12g --xmx 6g --cache 5120m --executors 24 --iothreads 24
It’s ok now to generate the startup script while fail to start the daemon due to:
AMRMClientAsync.onError() received org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=12288, maxMemory=8192
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:268)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:228)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:244)
at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106)
at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:502)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
In this example, I want to use 6g for container running memory with 5g for off-heap data cache, may result to 12g memory in total. The example shows the ```size``` is taken over the ```xmx```. If i decrease the size to 6g, the llap daemon starts with 6g heap memory, which I see from the gc log in nodemanager web.
the hive version: apache 2.1.1
So do i miss anything? I get confused.
Thanks
Zhihua, Deng
在 2017年3月14日，下午12:35，Prasanth Jayachandran <pj...@hortonworks.com>> 写道：
This essentially means you are asking for a container of size 4GB and all of that 4GB is used for cache. There is no memory for executors.
Ideally you want to set, size = Xmx + cache (if off heap cache is used) + some headroom space.
Xmx - Xmx is the heap memory used by the executors
Cache - if direct allocation is used this will count towards off-heap memory usage
Headroom space - this is also off heap memory used for misc java related stuffs (metaspace, threads * stack size, gc data structures etc.)
Say if you are using 4GB containers and you want 2 executors with 1GB each,
then you can try the following for off heap cache
—size 4g —Xmx 2g —cache 1536m
for on-heap cache
—size 4g —Xmx 3584m
Remaining ~500MB is left as headroom so that YARN does not kill the container when it reaches the physical memory usage limits.
Reference usage can be found here https://github.com/t3rmin4t0r/tez-autobuild/blob/llap/slider-gen.sh
Thanks
Prasanth
On Mar 13, 2017, at 9:22 PM, 邓志华 <zh...@CREDITEASE.CN>> wrote:
when I use 'hive --service llap --name llap_service --size 4g --cache 4g' to generate the startup script for llap, an exception throws:
java.lang.IllegalArgumentException: Cache size (4.00GB) has to be smaller than the container sizing (4.00GB)
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
at org.apache.hadoop.hive.llap.cli.LlapServiceDriver.run(LlapServiceDriver.java:207)
at org.apache.hadoop.hive.llap.cli.LlapServiceDriver.main(LlapServiceDriver.java:104)
is there any reasons why should follow this rule?
Thanks
Zhihua Deng
Re: Why direct memory should be smaller than llap size
Posted by 邓志华 <zh...@CREDITEASE.CN>.
Thanks @pjayachandran, I try to add xmx to specify the executors memory.
hive --service llap --name llap_service --instances 16 --size 12g --xmx 6g --cache 5120m --executors 24 --iothreads 24
It’s ok now to generate the startup script while fail to start the daemon due to:
AMRMClientAsync.onError() received org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=12288, maxMemory=8192
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:268)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:228)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:244)
at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106)
at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:502)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
In this example, I want to use 6g for container running memory with 5g for off-heap data cache, may result to 12g memory in total. The example shows the ```size``` is taken over the ```xmx```. If i decrease the size to 6g, the llap daemon starts with 6g heap memory, which I see from the gc log in nodemanager web.
the hive version: apache 2.1.1
So do i miss anything? I get confused.
Thanks
Zhihua, Deng
在 2017年3月14日，下午12:35，Prasanth Jayachandran <pj...@hortonworks.com>> 写道：
This essentially means you are asking for a container of size 4GB and all of that 4GB is used for cache. There is no memory for executors.
Ideally you want to set, size = Xmx + cache (if off heap cache is used) + some headroom space.
Xmx - Xmx is the heap memory used by the executors
Cache - if direct allocation is used this will count towards off-heap memory usage
Headroom space - this is also off heap memory used for misc java related stuffs (metaspace, threads * stack size, gc data structures etc.)
Say if you are using 4GB containers and you want 2 executors with 1GB each,
then you can try the following for off heap cache
—size 4g —Xmx 2g —cache 1536m
for on-heap cache
—size 4g —Xmx 3584m
Remaining ~500MB is left as headroom so that YARN does not kill the container when it reaches the physical memory usage limits.
Reference usage can be found here https://github.com/t3rmin4t0r/tez-autobuild/blob/llap/slider-gen.sh
Thanks
Prasanth
On Mar 13, 2017, at 9:22 PM, 邓志华 <zh...@CREDITEASE.CN>> wrote:
when I use 'hive --service llap --name llap_service --size 4g --cache 4g' to generate the startup script for llap, an exception throws:
java.lang.IllegalArgumentException: Cache size (4.00GB) has to be smaller than the container sizing (4.00GB)
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
at org.apache.hadoop.hive.llap.cli.LlapServiceDriver.run(LlapServiceDriver.java:207)
at org.apache.hadoop.hive.llap.cli.LlapServiceDriver.main(LlapServiceDriver.java:104)
is there any reasons why should follow this rule?
Thanks
Zhihua Deng
Re: Why direct memory should be smaller than llap size
Posted by Prasanth Jayachandran <pj...@hortonworks.com>.
This essentially means you are asking for a container of size 4GB and all of that 4GB is used for cache. There is no memory for executors.
Ideally you want to set, size = Xmx + cache (if off heap cache is used) + some headroom space.
Xmx - Xmx is the heap memory used by the executors
Cache - if direct allocation is used this will count towards off-heap memory usage
Headroom space - this is also off heap memory used for misc java related stuffs (metaspace, threads * stack size, gc data structures etc.)
Say if you are using 4GB containers and you want 2 executors with 1GB each,
then you can try the following for off heap cache
—size 4g —Xmx 2g —cache 1536m
for on-heap cache
—size 4g —Xmx 3584m
Remaining ~500MB is left as headroom so that YARN does not kill the container when it reaches the physical memory usage limits.
Reference usage can be found here https://github.com/t3rmin4t0r/tez-autobuild/blob/llap/slider-gen.sh
Thanks
Prasanth
On Mar 13, 2017, at 9:22 PM, 邓志华 <zh...@CREDITEASE.CN>> wrote:
when I use 'hive --service llap --name llap_service --size 4g --cache 4g' to generate the startup script for llap, an exception throws:
java.lang.IllegalArgumentException: Cache size (4.00GB) has to be smaller than the container sizing (4.00GB)
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
at org.apache.hadoop.hive.llap.cli.LlapServiceDriver.run(LlapServiceDriver.java:207)
at org.apache.hadoop.hive.llap.cli.LlapServiceDriver.main(LlapServiceDriver.java:104)
is there any reasons why should follow this rule?
Thanks
Zhihua Deng
Why direct memory should be smaller than llap size
Posted by 邓志华 <zh...@CREDITEASE.CN>.
when I use 'hive --service llap --name llap_service --size 4g --cache 4g' to generate the startup script for llap, an exception throws:
java.lang.IllegalArgumentException: Cache size (4.00GB) has to be smaller than the container sizing (4.00GB)
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
at org.apache.hadoop.hive.llap.cli.LlapServiceDriver.run(LlapServiceDriver.java:207)
at org.apache.hadoop.hive.llap.cli.LlapServiceDriver.main(LlapServiceDriver.java:104)
is there any reasons why should follow this rule?
Thanks
Zhihua Deng
Re: Missing column headers from S3 file
Posted by Punit Shetty <pu...@gmail.com>.
Yes, running the CTAS command on hive CLI does print the headers once the
query execution is completed.
I think this feature is yet to be implemented: https://issues.
apache.org/jira/browse/HIVE-12860
On Thu, Mar 9, 2017 at 9:50 AM, Punit Shetty <pu...@gmail.com>
wrote:
> Yes, running the CTAS command on hive CLI does print the headers once the
> query execution is completed.
>
> I think this feature is yet to be implemented: https://issues.
> apache.org/jira/browse/HIVE-12860
>
>
> On Wed, Mar 8, 2017 at 5:51 PM, Pushkar.Gujar <pu...@gmail.com>
> wrote:
>
>> apologies,
>>
>> my command was
>>
>> hive -e "set hive.cli.print.header=true; *select * from abc*" >>
>> output.txt
>>
>>
>> Thank you,
>> *Pushkar Gujar*
>>
>>
>> On Wed, Mar 8, 2017 at 8:49 PM, Pushkar.Gujar <pu...@gmail.com>
>> wrote:
>>
>>> if you were running CTAS command from hive CLI, you might have noticed
>>> that headers get printed in CLI once the query execution is completed. I
>>> think the property hive.cli.print.header is there to only print headers
>>> only in cli.
>>>
>>> Not sure about S3, but I tried below which worked perfectly
>>>
>>> hive -e "set hive.cli.print.header=true; create table....." >> output.txt
>>>
>>>
>>> Thank you,
>>> *Pushkar Gujar*
>>>
>>>
>>> On Wed, Mar 8, 2017 at 6:47 PM, Punit Shetty <pu...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm trying to get the column headers into a S3 file.
>>>>
>>>> When I run this on the hive command line:
>>>> set hive.cli.print.header=true;
>>>> select * from abc;
>>>>
>>>> Things work fine, I'm able to see the column headers with the table
>>>> data on the console.
>>>>
>>>> But my query is, when I try something like this:
>>>> set hive.cli.print.header=true;
>>>> CREATE TABLE test123 ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
>>>> LINES TERMINATED BY '\n' STORED AS TEXTFILE LOCATION 's3://somelocation' AS
>>>> select * abc;
>>>>
>>>> The column headers are missing from the S3 file.
>>>>
>>>> Is this the expected behaviour or am I missing something?
>>>>
>>>> Thanks,
>>>> Punit
>>>>
>>>
>>>
>>
>
Re: Missing column headers from S3 file
Posted by "Pushkar.Gujar" <pu...@gmail.com>.
apologies,
my command was
hive -e "set hive.cli.print.header=true; *select * from abc*" >> output.txt
Thank you,
*Pushkar Gujar*
On Wed, Mar 8, 2017 at 8:49 PM, Pushkar.Gujar <pu...@gmail.com>
wrote:
> if you were running CTAS command from hive CLI, you might have noticed
> that headers get printed in CLI once the query execution is completed. I
> think the property hive.cli.print.header is there to only print headers
> only in cli.
>
> Not sure about S3, but I tried below which worked perfectly
>
> hive -e "set hive.cli.print.header=true; create table....." >> output.txt
>
>
> Thank you,
> *Pushkar Gujar*
>
>
> On Wed, Mar 8, 2017 at 6:47 PM, Punit Shetty <pu...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm trying to get the column headers into a S3 file.
>>
>> When I run this on the hive command line:
>> set hive.cli.print.header=true;
>> select * from abc;
>>
>> Things work fine, I'm able to see the column headers with the table data
>> on the console.
>>
>> But my query is, when I try something like this:
>> set hive.cli.print.header=true;
>> CREATE TABLE test123 ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES
>> TERMINATED BY '\n' STORED AS TEXTFILE LOCATION 's3://somelocation' AS
>> select * abc;
>>
>> The column headers are missing from the S3 file.
>>
>> Is this the expected behaviour or am I missing something?
>>
>> Thanks,
>> Punit
>>
>
>
Re: Missing column headers from S3 file
Posted by "Pushkar.Gujar" <pu...@gmail.com>.
if you were running CTAS command from hive CLI, you might have noticed that
headers get printed in CLI once the query execution is completed. I think
the property hive.cli.print.header is there to only print headers only in
cli.
Not sure about S3, but I tried below which worked perfectly
hive -e "set hive.cli.print.header=true; create table....." >> output.txt
Thank you,
*Pushkar Gujar*
On Wed, Mar 8, 2017 at 6:47 PM, Punit Shetty <pu...@gmail.com>
wrote:
> Hi,
>
> I'm trying to get the column headers into a S3 file.
>
> When I run this on the hive command line:
> set hive.cli.print.header=true;
> select * from abc;
>
> Things work fine, I'm able to see the column headers with the table data
> on the console.
>
> But my query is, when I try something like this:
> set hive.cli.print.header=true;
> CREATE TABLE test123 ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES
> TERMINATED BY '\n' STORED AS TEXTFILE LOCATION 's3://somelocation' AS
> select * abc;
>
> The column headers are missing from the S3 file.
>
> Is this the expected behaviour or am I missing something?
>
> Thanks,
> Punit
>