You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hawq.apache.org by Shubham Sharma <ss...@pivotal.io> on 2017/11/14 01:25:48 UTC
HAWQ-1548 - Ambiguous message while logging hawq utilization
While YARN mode is enabled, resource broker logs two things -
- YARN cluster total resource
- HAWQ's total resource per node.
Following messages are logged
2017-11-11 23:21:40.944904
UTC,,,p549330,th900077856,,,,0,con4,,seg-10000,,,,,"LOG","00000","Resource
manager YARN resource broker counted YARN cluster having total
resource (1376256 MB, 168.000000
CORE).",,,,,,,0,,"resourcebroker_LIBYARN.c",776,
2017-11-11 23:21:40.944921
UTC,,,p549330,th900077856,,,,0,con4,,seg-10000,,,,,"LOG","00000","Resource
manager YARN resource broker counted HAWQ cluster now having (98304
MB, 12.000000 CORE) in a YARN cluster of total resource (1376256 MB,
168.000000 CORE).",,,,,,,0,,"resourcebroker_LIBYARN.c",785,
The second message shown above is ambiguous, After reading the sentence
below it looks like that complete Hawq cluster in whole has only 98304 MB
and 12 cores. However according to the configuration it should be 98304 MB
and 12 cores per segment server.
Resource manager YARN resource broker counted HAWQ cluster now having
(98304 MB, 12.000000 CORE) in a YARN cluster of total resource
(1376256 MB, 168.000000 CORE).
Either the wrong variables are printed or we can correct the message to
represent that the resources logged are per node. As this can confuse the
user into thinking that hawq cluster does not have enough resources.
Is this understanding correct ?
Regards,
Shubham Sharma
Re: HAWQ-1548 - Ambiguous message while logging hawq utilization
Posted by Shubham Sharma <ss...@pivotal.io>.
Here is the exact resource_manager_pool_status
/tmp/resource_manager_respool_status
HOST_ID(id=0:hostname:segment06.gcp.test-domain.com)
HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12)
HOST_AVAILABLITY(HAWQAvailable=true)
HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595756349163:RUAlivePending=false)
HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000)
HOST_ID(id=1:hostname:segment11.gcp.test-domain.com)
HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12)
HOST_AVAILABLITY(HAWQAvailable=true)
HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595756544067:RUAlivePending=false)
HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000)
HOST_ID(id=2:hostname:segment12.gcp.test-domain.com)
HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12)
HOST_AVAILABLITY(HAWQAvailable=true)
HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595756749623:RUAlivePending=false)
HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000)
HOST_ID(id=3:hostname:segment05.gcp.test-domain.com)
HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12)
HOST_AVAILABLITY(HAWQAvailable=true)
HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595756246999:RUAlivePending=false)
HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000)
HOST_ID(id=4:hostname:segment10.gcp.test-domain.com)
HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12)
HOST_AVAILABLITY(HAWQAvailable=true)
HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595757552955:RUAlivePending=false)
HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000)
HOST_ID(id=5:hostname:segment03.gcp.test-domain.com)
HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12)
HOST_AVAILABLITY(HAWQAvailable=true)
HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595758356042:RUAlivePending=false)
HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000)
HOST_ID(id=6:hostname:segment09.gcp.test-domain.com)
HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12)
HOST_AVAILABLITY(HAWQAvailable=true)
HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595759258665:RUAlivePending=false)
HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000)
HOST_ID(id=7:hostname:segment04.gcp.test-domain.com)
HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12)
HOST_AVAILABLITY(HAWQAvailable=true)
HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595754750573:RUAlivePending=false)
HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000)
HOST_ID(id=8:hostname:segment02.gcp.test-domain.com)
HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12)
HOST_AVAILABLITY(HAWQAvailable=true)
HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595758451056:RUAlivePending=false)
HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000)
HOST_ID(id=9:hostname:segment13.gcp.test-domain.com)
HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12)
HOST_AVAILABLITY(HAWQAvailable=true)
HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595758958073:RUAlivePending=false)
HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000)
HOST_ID(id=10:hostname:segment08.gcp.test-domain.com)
HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12)
HOST_AVAILABLITY(HAWQAvailable=true)
HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595759458370:RUAlivePending=false)
HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000)
HOST_ID(id=11:hostname:segment07.gcp.test-domain.com)
HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12)
HOST_AVAILABLITY(HAWQAvailable=true)
HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595741128453:RUAlivePending=false)
HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000)
HOST_ID(id=12:hostname:segment01.gcp.test-domain.com)
HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12)
HOST_AVAILABLITY(HAWQAvailable=true)
HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595737023537:RUAlivePending=false)
HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000)
HOST_ID(id=13:hostname:segment14.gcp.test-domain.com)
HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12)
HOST_AVAILABLITY(HAWQAvailable=true)
HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595735919485:RUAlivePending=false)
HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000)
On Mon, Nov 13, 2017 at 8:10 PM, Shubham Sharma <ss...@pivotal.io> wrote:
Hi Wen,
>
> I thought the same but when I look at - resource_manager_respool_status
> using - SELECT * FROM dump_resource_manager_status(2); All segments are
> registered and each has configuration 98304MB and 12.000000 CORE
>
> On Mon, Nov 13, 2017 at 7:40 PM, Wen Lin <wl...@pivotal.io> wrote:
>
>> Hi, Shubham,
>>
>> The second message means that, this YARN cluster's total resource is
>> 1376256
>> MB, 168.000000 CORE.
>> And HAWQ clusters have the resource: 98304MB, 12.000000 CORE of the whole
>> YARN cluster now.
>> The other resources in YARN haven't been allocated for HAWQ.
>> Maybe there are no HAWQ segments running on these segments at the moment,
>> or other reasons.
>>
>> Thanks!
>>
>>
>> On Tue, Nov 14, 2017 at 9:25 AM, Shubham Sharma <ss...@pivotal.io>
>> wrote:
>>
>> > While YARN mode is enabled, resource broker logs two things -
>> >
>> > - YARN cluster total resource
>> > - HAWQ's total resource per node.
>> >
>> > Following messages are logged
>> >
>> > 2017-11-11 23:21:40.944904
>> > UTC,,,p549330,th900077856,,,,0,con4,,seg-10000,,,,,"LOG","00
>> 000","Resource
>> > manager YARN resource broker counted YARN cluster having total
>> > resource (1376256 MB, 168.000000
>> > CORE).",,,,,,,0,,"resourcebroker_LIBYARN.c",776,
>> >
>> > 2017-11-11 23:21:40.944921
>> > UTC,,,p549330,th900077856,,,,0,con4,,seg-10000,,,,,"LOG","00
>> 000","Resource
>> > manager YARN resource broker counted HAWQ cluster now having (98304
>> > MB, 12.000000 CORE) in a YARN cluster of total resource (1376256 MB,
>> > 168.000000 CORE).",,,,,,,0,,"resourcebroker_LIBYARN.c",785,
>> >
>> > The second message shown above is ambiguous, After reading the sentence
>> > below it looks like that complete Hawq cluster in whole has only 98304
>> MB
>> > and 12 cores. However according to the configuration it should be 98304
>> MB
>> > and 12 cores per segment server.
>> >
>> > Resource manager YARN resource broker counted HAWQ cluster now having
>> > (98304 MB, 12.000000 CORE) in a YARN cluster of total resource
>> > (1376256 MB, 168.000000 CORE).
>> >
>> > Either the wrong variables are printed or we can correct the message to
>> > represent that the resources logged are per node. As this can confuse
>> the
>> > user into thinking that hawq cluster does not have enough resources.
>> >
>> > Is this understanding correct ?
>> >
>> > Regards,
>> > Shubham Sharma
>> >
>>
>
>
>
> --
> Regards,
> Shubham Sharma
> Staff Customer Engineer
> Pivotal Global Support Services
> ssharma@pivotal.io
> Direct Tel: +1(510)-304-8201 <(510)%20304-8201>
> Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
> Out of Office Hours Contact +1 877-477-2269 <(877)%20477-2269>
>
--
Regards,
Shubham Sharma
Staff Customer Engineer
Pivotal Global Support Services
ssharma@pivotal.io
Direct Tel: +1(510)-304-8201 <(510)%20304-8201>
Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
Out of Office Hours Contact +1 877-477-2269 <(877)%20477-2269>
Re: HAWQ-1548 - Ambiguous message while logging hawq utilization
Posted by Shubham Sharma <ss...@pivotal.io>.
Hi Wen,
I thought the same but when I look at - resource_manager_respool_status
using - SELECT * FROM dump_resource_manager_status(2); All segments are
registered and each has configuration 98304MB and 12.000000 CORE
On Mon, Nov 13, 2017 at 7:40 PM, Wen Lin <wl...@pivotal.io> wrote:
> Hi, Shubham,
>
> The second message means that, this YARN cluster's total resource is
> 1376256
> MB, 168.000000 CORE.
> And HAWQ clusters have the resource: 98304MB, 12.000000 CORE of the whole
> YARN cluster now.
> The other resources in YARN haven't been allocated for HAWQ.
> Maybe there are no HAWQ segments running on these segments at the moment,
> or other reasons.
>
> Thanks!
>
>
> On Tue, Nov 14, 2017 at 9:25 AM, Shubham Sharma <ss...@pivotal.io>
> wrote:
>
> > While YARN mode is enabled, resource broker logs two things -
> >
> > - YARN cluster total resource
> > - HAWQ's total resource per node.
> >
> > Following messages are logged
> >
> > 2017-11-11 23:21:40.944904
> > UTC,,,p549330,th900077856,,,,0,con4,,seg-10000,,,,,"LOG","
> 00000","Resource
> > manager YARN resource broker counted YARN cluster having total
> > resource (1376256 MB, 168.000000
> > CORE).",,,,,,,0,,"resourcebroker_LIBYARN.c",776,
> >
> > 2017-11-11 23:21:40.944921
> > UTC,,,p549330,th900077856,,,,0,con4,,seg-10000,,,,,"LOG","
> 00000","Resource
> > manager YARN resource broker counted HAWQ cluster now having (98304
> > MB, 12.000000 CORE) in a YARN cluster of total resource (1376256 MB,
> > 168.000000 CORE).",,,,,,,0,,"resourcebroker_LIBYARN.c",785,
> >
> > The second message shown above is ambiguous, After reading the sentence
> > below it looks like that complete Hawq cluster in whole has only 98304 MB
> > and 12 cores. However according to the configuration it should be 98304
> MB
> > and 12 cores per segment server.
> >
> > Resource manager YARN resource broker counted HAWQ cluster now having
> > (98304 MB, 12.000000 CORE) in a YARN cluster of total resource
> > (1376256 MB, 168.000000 CORE).
> >
> > Either the wrong variables are printed or we can correct the message to
> > represent that the resources logged are per node. As this can confuse the
> > user into thinking that hawq cluster does not have enough resources.
> >
> > Is this understanding correct ?
> >
> > Regards,
> > Shubham Sharma
> >
>
--
Regards,
Shubham Sharma
Staff Customer Engineer
Pivotal Global Support Services
ssharma@pivotal.io
Direct Tel: +1(510)-304-8201
Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
Out of Office Hours Contact +1 877-477-2269
Re: HAWQ-1548 - Ambiguous message while logging hawq utilization
Posted by Wen Lin <wl...@pivotal.io>.
Hi, Shubham,
The second message means that, this YARN cluster's total resource is 1376256
MB, 168.000000 CORE.
And HAWQ clusters have the resource: 98304MB, 12.000000 CORE of the whole
YARN cluster now.
The other resources in YARN haven't been allocated for HAWQ.
Maybe there are no HAWQ segments running on these segments at the moment,
or other reasons.
Thanks!
On Tue, Nov 14, 2017 at 9:25 AM, Shubham Sharma <ss...@pivotal.io> wrote:
> While YARN mode is enabled, resource broker logs two things -
>
> - YARN cluster total resource
> - HAWQ's total resource per node.
>
> Following messages are logged
>
> 2017-11-11 23:21:40.944904
> UTC,,,p549330,th900077856,,,,0,con4,,seg-10000,,,,,"LOG","00000","Resource
> manager YARN resource broker counted YARN cluster having total
> resource (1376256 MB, 168.000000
> CORE).",,,,,,,0,,"resourcebroker_LIBYARN.c",776,
>
> 2017-11-11 23:21:40.944921
> UTC,,,p549330,th900077856,,,,0,con4,,seg-10000,,,,,"LOG","00000","Resource
> manager YARN resource broker counted HAWQ cluster now having (98304
> MB, 12.000000 CORE) in a YARN cluster of total resource (1376256 MB,
> 168.000000 CORE).",,,,,,,0,,"resourcebroker_LIBYARN.c",785,
>
> The second message shown above is ambiguous, After reading the sentence
> below it looks like that complete Hawq cluster in whole has only 98304 MB
> and 12 cores. However according to the configuration it should be 98304 MB
> and 12 cores per segment server.
>
> Resource manager YARN resource broker counted HAWQ cluster now having
> (98304 MB, 12.000000 CORE) in a YARN cluster of total resource
> (1376256 MB, 168.000000 CORE).
>
> Either the wrong variables are printed or we can correct the message to
> represent that the resources logged are per node. As this can confuse the
> user into thinking that hawq cluster does not have enough resources.
>
> Is this understanding correct ?
>
> Regards,
> Shubham Sharma
>