You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Krishna Kishore Bonagiri <wr...@gmail.com> on 2013/08/01 11:19:07 UTC
Re: Extra start-up overhead with hadoop-2.1.0-beta
Hi,
Please share with me if you anyone has an answer or clues to my question
regarding the start up performance.
Also, one more thing I have observed today is the time taken to run a
command on a container went up by more than a second in this latest version.
When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point I
call startContainer() to the point the command is started on the container.
where as
When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
came to the call back onContainerStarted() to the point the command is seen
started running on the container.
Thanks,
Kishore
On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi,
>
> I have been using the hadoop-2.0.1-beta release candidate and observed
> that it is slower in running my simple application that runs on 2
> containers. I have tried to find out which parts of it is really having
> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
> found that.
>
> 1) From the point my Client has submitted the Application Master to RM, it
> is taking 2 seconds extra
> 2) From the point my container request are set up by Application Master,
> till the containers are allocated, it is taking 2 seconds extra
>
> Is this overhead expected with the changes that went into the new version?
> Or is there to improve it by changing something in configurations or so?
>
> Thanks,
> Kishore
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
No Ravi, I am not running any MR job. Also, my configuration files are not
big.
On Wed, Aug 7, 2013 at 11:12 PM, Ravi Prakash <ra...@ymail.com> wrote:
> I believe https://issues.apache.org/jira/browse/MAPREDUCE-5399 causes
> performance degradation in cases where there are a lot of reducers. I can
> imagine it causing degradation if the configuration files are super big /
> some other weird cases.
>
>
> ------------------------------
> *From:* Krishna Kishore Bonagiri <wr...@gmail.com>
> *To:* user@hadoop.apache.org
> *Sent:* Wednesday, August 7, 2013 10:03 AM
> *Subject:* Re: Extra start-up overhead with hadoop-2.1.0-beta
>
> Hi Omkar,
>
> Can you please see if you can answer my question with this info or if you
> need anything else from me?
>
> Also, does resource localization improve or impact any performance?
>
> Thanks,
> Kishore
>
>
> On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>
> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
>
> More information will help a lot.
>
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com/>
>
>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
> Hi,
> Please share with me if you anyone has an answer or clues to my question
> regarding the start up performance.
>
> Also, one more thing I have observed today is the time taken to run a
> command on a container went up by more than a second in this latest version.
>
> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
> I call startContainer() to the point the command is started on the
> container.
>
> where as
>
> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
> came to the call back onContainerStarted() to the point the command is seen
> started running on the container.
>
> Thanks,
> Kishore
>
>
> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
> Hi,
>
> I have been using the hadoop-2.0.1-beta release candidate and observed
> that it is slower in running my simple application that runs on 2
> containers. I have tried to find out which parts of it is really having
> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
> found that.
>
> 1) From the point my Client has submitted the Application Master to RM, it
> is taking 2 seconds extra
> 2) From the point my container request are set up by Application Master,
> till the containers are allocated, it is taking 2 seconds extra
>
> Is this overhead expected with the changes that went into the new version?
> Or is there to improve it by changing something in configurations or so?
>
> Thanks,
> Kishore
>
>
>
>
>
>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
No Ravi, I am not running any MR job. Also, my configuration files are not
big.
On Wed, Aug 7, 2013 at 11:12 PM, Ravi Prakash <ra...@ymail.com> wrote:
> I believe https://issues.apache.org/jira/browse/MAPREDUCE-5399 causes
> performance degradation in cases where there are a lot of reducers. I can
> imagine it causing degradation if the configuration files are super big /
> some other weird cases.
>
>
> ------------------------------
> *From:* Krishna Kishore Bonagiri <wr...@gmail.com>
> *To:* user@hadoop.apache.org
> *Sent:* Wednesday, August 7, 2013 10:03 AM
> *Subject:* Re: Extra start-up overhead with hadoop-2.1.0-beta
>
> Hi Omkar,
>
> Can you please see if you can answer my question with this info or if you
> need anything else from me?
>
> Also, does resource localization improve or impact any performance?
>
> Thanks,
> Kishore
>
>
> On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>
> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
>
> More information will help a lot.
>
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com/>
>
>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
> Hi,
> Please share with me if you anyone has an answer or clues to my question
> regarding the start up performance.
>
> Also, one more thing I have observed today is the time taken to run a
> command on a container went up by more than a second in this latest version.
>
> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
> I call startContainer() to the point the command is started on the
> container.
>
> where as
>
> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
> came to the call back onContainerStarted() to the point the command is seen
> started running on the container.
>
> Thanks,
> Kishore
>
>
> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
> Hi,
>
> I have been using the hadoop-2.0.1-beta release candidate and observed
> that it is slower in running my simple application that runs on 2
> containers. I have tried to find out which parts of it is really having
> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
> found that.
>
> 1) From the point my Client has submitted the Application Master to RM, it
> is taking 2 seconds extra
> 2) From the point my container request are set up by Application Master,
> till the containers are allocated, it is taking 2 seconds extra
>
> Is this overhead expected with the changes that went into the new version?
> Or is there to improve it by changing something in configurations or so?
>
> Thanks,
> Kishore
>
>
>
>
>
>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
No Ravi, I am not running any MR job. Also, my configuration files are not
big.
On Wed, Aug 7, 2013 at 11:12 PM, Ravi Prakash <ra...@ymail.com> wrote:
> I believe https://issues.apache.org/jira/browse/MAPREDUCE-5399 causes
> performance degradation in cases where there are a lot of reducers. I can
> imagine it causing degradation if the configuration files are super big /
> some other weird cases.
>
>
> ------------------------------
> *From:* Krishna Kishore Bonagiri <wr...@gmail.com>
> *To:* user@hadoop.apache.org
> *Sent:* Wednesday, August 7, 2013 10:03 AM
> *Subject:* Re: Extra start-up overhead with hadoop-2.1.0-beta
>
> Hi Omkar,
>
> Can you please see if you can answer my question with this info or if you
> need anything else from me?
>
> Also, does resource localization improve or impact any performance?
>
> Thanks,
> Kishore
>
>
> On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>
> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
>
> More information will help a lot.
>
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com/>
>
>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
> Hi,
> Please share with me if you anyone has an answer or clues to my question
> regarding the start up performance.
>
> Also, one more thing I have observed today is the time taken to run a
> command on a container went up by more than a second in this latest version.
>
> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
> I call startContainer() to the point the command is started on the
> container.
>
> where as
>
> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
> came to the call back onContainerStarted() to the point the command is seen
> started running on the container.
>
> Thanks,
> Kishore
>
>
> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
> Hi,
>
> I have been using the hadoop-2.0.1-beta release candidate and observed
> that it is slower in running my simple application that runs on 2
> containers. I have tried to find out which parts of it is really having
> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
> found that.
>
> 1) From the point my Client has submitted the Application Master to RM, it
> is taking 2 seconds extra
> 2) From the point my container request are set up by Application Master,
> till the containers are allocated, it is taking 2 seconds extra
>
> Is this overhead expected with the changes that went into the new version?
> Or is there to improve it by changing something in configurations or so?
>
> Thanks,
> Kishore
>
>
>
>
>
>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
No Ravi, I am not running any MR job. Also, my configuration files are not
big.
On Wed, Aug 7, 2013 at 11:12 PM, Ravi Prakash <ra...@ymail.com> wrote:
> I believe https://issues.apache.org/jira/browse/MAPREDUCE-5399 causes
> performance degradation in cases where there are a lot of reducers. I can
> imagine it causing degradation if the configuration files are super big /
> some other weird cases.
>
>
> ------------------------------
> *From:* Krishna Kishore Bonagiri <wr...@gmail.com>
> *To:* user@hadoop.apache.org
> *Sent:* Wednesday, August 7, 2013 10:03 AM
> *Subject:* Re: Extra start-up overhead with hadoop-2.1.0-beta
>
> Hi Omkar,
>
> Can you please see if you can answer my question with this info or if you
> need anything else from me?
>
> Also, does resource localization improve or impact any performance?
>
> Thanks,
> Kishore
>
>
> On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com>wrote:
>
> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
>
> More information will help a lot.
>
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com/>
>
>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
> Hi,
> Please share with me if you anyone has an answer or clues to my question
> regarding the start up performance.
>
> Also, one more thing I have observed today is the time taken to run a
> command on a container went up by more than a second in this latest version.
>
> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
> I call startContainer() to the point the command is started on the
> container.
>
> where as
>
> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
> came to the call back onContainerStarted() to the point the command is seen
> started running on the container.
>
> Thanks,
> Kishore
>
>
> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
> Hi,
>
> I have been using the hadoop-2.0.1-beta release candidate and observed
> that it is slower in running my simple application that runs on 2
> containers. I have tried to find out which parts of it is really having
> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
> found that.
>
> 1) From the point my Client has submitted the Application Master to RM, it
> is taking 2 seconds extra
> 2) From the point my container request are set up by Application Master,
> till the containers are allocated, it is taking 2 seconds extra
>
> Is this overhead expected with the changes that went into the new version?
> Or is there to improve it by changing something in configurations or so?
>
> Thanks,
> Kishore
>
>
>
>
>
>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Ravi Prakash <ra...@ymail.com>.
I believe https://issues.apache.org/jira/browse/MAPREDUCE-5399 causes performance degradation in cases where there are a lot of reducers. I can imagine it causing degradation if the configuration files are super big / some other weird cases.
________________________________
From: Krishna Kishore Bonagiri <wr...@gmail.com>
To: user@hadoop.apache.org
Sent: Wednesday, August 7, 2013 10:03 AM
Subject: Re: Extra start-up overhead with hadoop-2.1.0-beta
Hi Omkar,
Can you please see if you can answer my question with this info or if you need anything else from me?
Also, does resource localization improve or impact any performance?
Thanks,
Kishore
On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
How are you making these measurements can you elaborate more? Is it on a best case basis or on an average or worst case? How many resources are you sending it for localization? were the sizes and number of these resources consistent across tests? Were these resources public/private/application specific? Apart from this is the other load on node manager same? is the load on hdfs same? did you see any network bottleneck?
>
>
>More information will help a lot.
>
>
>
>
>
>Thanks,
>Omkar Joshi
>Hortonworks Inc.
>
>
>
>On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
>
>Hi,
>> Please share with me if you anyone has an answer or clues to my question regarding the start up performance.
>>
>>
>>Also, one more thing I have observed today is the time taken to run a command on a container went up by more than a second in this latest version.
>>
>>
>>When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point I call startContainer() to the point the command is started on the container.
>>
>>
>>where as
>>
>>
>>When using 2.1.0-beta, it is taking around 1.5 seconds from the point it came to the call back onContainerStarted() to the point the command is seen started running on the container.
>>
>>
>>Thanks,
>>
>>Kishore
>>
>>
>>
>>On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
>>
>>Hi,
>>>
>>>
>>> I have been using the hadoop-2.0.1-beta release candidate and observed that it is slower in running my simple application that runs on 2 containers. I have tried to find out which parts of it is really having this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I found that.
>>>
>>>
>>>1) From the point my Client has submitted the Application Master to RM, it is taking 2 seconds extra
>>>2) From the point my container request are set up by Application Master, till the containers are allocated, it is taking 2 seconds extra
>>>
>>>
>>>Is this overhead expected with the changes that went into the new version? Or is there to improve it by changing something in configurations or so?
>>>
>>>
>>>Thanks,
>>>Kishore
>>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Ravi Prakash <ra...@ymail.com>.
I believe https://issues.apache.org/jira/browse/MAPREDUCE-5399 causes performance degradation in cases where there are a lot of reducers. I can imagine it causing degradation if the configuration files are super big / some other weird cases.
________________________________
From: Krishna Kishore Bonagiri <wr...@gmail.com>
To: user@hadoop.apache.org
Sent: Wednesday, August 7, 2013 10:03 AM
Subject: Re: Extra start-up overhead with hadoop-2.1.0-beta
Hi Omkar,
Can you please see if you can answer my question with this info or if you need anything else from me?
Also, does resource localization improve or impact any performance?
Thanks,
Kishore
On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
How are you making these measurements can you elaborate more? Is it on a best case basis or on an average or worst case? How many resources are you sending it for localization? were the sizes and number of these resources consistent across tests? Were these resources public/private/application specific? Apart from this is the other load on node manager same? is the load on hdfs same? did you see any network bottleneck?
>
>
>More information will help a lot.
>
>
>
>
>
>Thanks,
>Omkar Joshi
>Hortonworks Inc.
>
>
>
>On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
>
>Hi,
>> Please share with me if you anyone has an answer or clues to my question regarding the start up performance.
>>
>>
>>Also, one more thing I have observed today is the time taken to run a command on a container went up by more than a second in this latest version.
>>
>>
>>When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point I call startContainer() to the point the command is started on the container.
>>
>>
>>where as
>>
>>
>>When using 2.1.0-beta, it is taking around 1.5 seconds from the point it came to the call back onContainerStarted() to the point the command is seen started running on the container.
>>
>>
>>Thanks,
>>
>>Kishore
>>
>>
>>
>>On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
>>
>>Hi,
>>>
>>>
>>> I have been using the hadoop-2.0.1-beta release candidate and observed that it is slower in running my simple application that runs on 2 containers. I have tried to find out which parts of it is really having this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I found that.
>>>
>>>
>>>1) From the point my Client has submitted the Application Master to RM, it is taking 2 seconds extra
>>>2) From the point my container request are set up by Application Master, till the containers are allocated, it is taking 2 seconds extra
>>>
>>>
>>>Is this overhead expected with the changes that went into the new version? Or is there to improve it by changing something in configurations or so?
>>>
>>>
>>>Thanks,
>>>Kishore
>>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Ravi Prakash <ra...@ymail.com>.
I believe https://issues.apache.org/jira/browse/MAPREDUCE-5399 causes performance degradation in cases where there are a lot of reducers. I can imagine it causing degradation if the configuration files are super big / some other weird cases.
________________________________
From: Krishna Kishore Bonagiri <wr...@gmail.com>
To: user@hadoop.apache.org
Sent: Wednesday, August 7, 2013 10:03 AM
Subject: Re: Extra start-up overhead with hadoop-2.1.0-beta
Hi Omkar,
Can you please see if you can answer my question with this info or if you need anything else from me?
Also, does resource localization improve or impact any performance?
Thanks,
Kishore
On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
How are you making these measurements can you elaborate more? Is it on a best case basis or on an average or worst case? How many resources are you sending it for localization? were the sizes and number of these resources consistent across tests? Were these resources public/private/application specific? Apart from this is the other load on node manager same? is the load on hdfs same? did you see any network bottleneck?
>
>
>More information will help a lot.
>
>
>
>
>
>Thanks,
>Omkar Joshi
>Hortonworks Inc.
>
>
>
>On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
>
>Hi,
>> Please share with me if you anyone has an answer or clues to my question regarding the start up performance.
>>
>>
>>Also, one more thing I have observed today is the time taken to run a command on a container went up by more than a second in this latest version.
>>
>>
>>When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point I call startContainer() to the point the command is started on the container.
>>
>>
>>where as
>>
>>
>>When using 2.1.0-beta, it is taking around 1.5 seconds from the point it came to the call back onContainerStarted() to the point the command is seen started running on the container.
>>
>>
>>Thanks,
>>
>>Kishore
>>
>>
>>
>>On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
>>
>>Hi,
>>>
>>>
>>> I have been using the hadoop-2.0.1-beta release candidate and observed that it is slower in running my simple application that runs on 2 containers. I have tried to find out which parts of it is really having this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I found that.
>>>
>>>
>>>1) From the point my Client has submitted the Application Master to RM, it is taking 2 seconds extra
>>>2) From the point my container request are set up by Application Master, till the containers are allocated, it is taking 2 seconds extra
>>>
>>>
>>>Is this overhead expected with the changes that went into the new version? Or is there to improve it by changing something in configurations or so?
>>>
>>>
>>>Thanks,
>>>Kishore
>>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Ravi Prakash <ra...@ymail.com>.
I believe https://issues.apache.org/jira/browse/MAPREDUCE-5399 causes performance degradation in cases where there are a lot of reducers. I can imagine it causing degradation if the configuration files are super big / some other weird cases.
________________________________
From: Krishna Kishore Bonagiri <wr...@gmail.com>
To: user@hadoop.apache.org
Sent: Wednesday, August 7, 2013 10:03 AM
Subject: Re: Extra start-up overhead with hadoop-2.1.0-beta
Hi Omkar,
Can you please see if you can answer my question with this info or if you need anything else from me?
Also, does resource localization improve or impact any performance?
Thanks,
Kishore
On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
How are you making these measurements can you elaborate more? Is it on a best case basis or on an average or worst case? How many resources are you sending it for localization? were the sizes and number of these resources consistent across tests? Were these resources public/private/application specific? Apart from this is the other load on node manager same? is the load on hdfs same? did you see any network bottleneck?
>
>
>More information will help a lot.
>
>
>
>
>
>Thanks,
>Omkar Joshi
>Hortonworks Inc.
>
>
>
>On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
>
>Hi,
>> Please share with me if you anyone has an answer or clues to my question regarding the start up performance.
>>
>>
>>Also, one more thing I have observed today is the time taken to run a command on a container went up by more than a second in this latest version.
>>
>>
>>When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point I call startContainer() to the point the command is started on the container.
>>
>>
>>where as
>>
>>
>>When using 2.1.0-beta, it is taking around 1.5 seconds from the point it came to the call back onContainerStarted() to the point the command is seen started running on the container.
>>
>>
>>Thanks,
>>
>>Kishore
>>
>>
>>
>>On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
>>
>>Hi,
>>>
>>>
>>> I have been using the hadoop-2.0.1-beta release candidate and observed that it is slower in running my simple application that runs on 2 containers. I have tried to find out which parts of it is really having this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I found that.
>>>
>>>
>>>1) From the point my Client has submitted the Application Master to RM, it is taking 2 seconds extra
>>>2) From the point my container request are set up by Application Master, till the containers are allocated, it is taking 2 seconds extra
>>>
>>>
>>>Is this overhead expected with the changes that went into the new version? Or is there to improve it by changing something in configurations or so?
>>>
>>>
>>>Thanks,
>>>Kishore
>>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Omkar,
Can you please see if you can answer my question with this info or if you
need anything else from me?
Also, does resource localization improve or impact any performance?
Thanks,
Kishore
On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
>
> More information will help a lot.
>
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>> Please share with me if you anyone has an answer or clues to my
>> question regarding the start up performance.
>>
>> Also, one more thing I have observed today is the time taken to run a
>> command on a container went up by more than a second in this latest version.
>>
>> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
>> I call startContainer() to the point the command is started on the
>> container.
>>
>> where as
>>
>> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
>> came to the call back onContainerStarted() to the point the command is seen
>> started running on the container.
>>
>> Thanks,
>> Kishore
>>
>>
>> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have been using the hadoop-2.0.1-beta release candidate and observed
>>> that it is slower in running my simple application that runs on 2
>>> containers. I have tried to find out which parts of it is really having
>>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
>>> found that.
>>>
>>> 1) From the point my Client has submitted the Application Master to RM,
>>> it is taking 2 seconds extra
>>> 2) From the point my container request are set up by Application Master,
>>> till the containers are allocated, it is taking 2 seconds extra
>>>
>>> Is this overhead expected with the changes that went into the new
>>> version? Or is there to improve it by changing something in configurations
>>> or so?
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Omkar,
I have got these number by running a simple C program on the containers
that fetches the timestamp in microseconds and exits. The times mentioned
are low and high, they are not varying so drastically with in a version but
there are huge differences(like a second) between the two versions,
2.0.4-alpha and 2.1.0-beta as I mentioned.
I am using a single node cluster, and all there is absolutely no other
load on the machine/node. My single node cluster is just used for my own
development work, and testing.
I am not aware of what is resource localization, I am not doing anything
specially for that.
Please let me know if you need any other info.
Thanks,
Kishore
On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
>
> More information will help a lot.
>
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>> Please share with me if you anyone has an answer or clues to my
>> question regarding the start up performance.
>>
>> Also, one more thing I have observed today is the time taken to run a
>> command on a container went up by more than a second in this latest version.
>>
>> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
>> I call startContainer() to the point the command is started on the
>> container.
>>
>> where as
>>
>> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
>> came to the call back onContainerStarted() to the point the command is seen
>> started running on the container.
>>
>> Thanks,
>> Kishore
>>
>>
>> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have been using the hadoop-2.0.1-beta release candidate and observed
>>> that it is slower in running my simple application that runs on 2
>>> containers. I have tried to find out which parts of it is really having
>>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
>>> found that.
>>>
>>> 1) From the point my Client has submitted the Application Master to RM,
>>> it is taking 2 seconds extra
>>> 2) From the point my container request are set up by Application Master,
>>> till the containers are allocated, it is taking 2 seconds extra
>>>
>>> Is this overhead expected with the changes that went into the new
>>> version? Or is there to improve it by changing something in configurations
>>> or so?
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Omkar,
Can you please see if you can answer my question with this info or if you
need anything else from me?
Also, does resource localization improve or impact any performance?
Thanks,
Kishore
On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
>
> More information will help a lot.
>
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>> Please share with me if you anyone has an answer or clues to my
>> question regarding the start up performance.
>>
>> Also, one more thing I have observed today is the time taken to run a
>> command on a container went up by more than a second in this latest version.
>>
>> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
>> I call startContainer() to the point the command is started on the
>> container.
>>
>> where as
>>
>> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
>> came to the call back onContainerStarted() to the point the command is seen
>> started running on the container.
>>
>> Thanks,
>> Kishore
>>
>>
>> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have been using the hadoop-2.0.1-beta release candidate and observed
>>> that it is slower in running my simple application that runs on 2
>>> containers. I have tried to find out which parts of it is really having
>>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
>>> found that.
>>>
>>> 1) From the point my Client has submitted the Application Master to RM,
>>> it is taking 2 seconds extra
>>> 2) From the point my container request are set up by Application Master,
>>> till the containers are allocated, it is taking 2 seconds extra
>>>
>>> Is this overhead expected with the changes that went into the new
>>> version? Or is there to improve it by changing something in configurations
>>> or so?
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Omkar,
Can you please see if you can answer my question with this info or if you
need anything else from me?
Also, does resource localization improve or impact any performance?
Thanks,
Kishore
On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
>
> More information will help a lot.
>
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>> Please share with me if you anyone has an answer or clues to my
>> question regarding the start up performance.
>>
>> Also, one more thing I have observed today is the time taken to run a
>> command on a container went up by more than a second in this latest version.
>>
>> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
>> I call startContainer() to the point the command is started on the
>> container.
>>
>> where as
>>
>> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
>> came to the call back onContainerStarted() to the point the command is seen
>> started running on the container.
>>
>> Thanks,
>> Kishore
>>
>>
>> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have been using the hadoop-2.0.1-beta release candidate and observed
>>> that it is slower in running my simple application that runs on 2
>>> containers. I have tried to find out which parts of it is really having
>>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
>>> found that.
>>>
>>> 1) From the point my Client has submitted the Application Master to RM,
>>> it is taking 2 seconds extra
>>> 2) From the point my container request are set up by Application Master,
>>> till the containers are allocated, it is taking 2 seconds extra
>>>
>>> Is this overhead expected with the changes that went into the new
>>> version? Or is there to improve it by changing something in configurations
>>> or so?
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Omkar,
Can you please see if you can answer my question with this info or if you
need anything else from me?
Also, does resource localization improve or impact any performance?
Thanks,
Kishore
On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
>
> More information will help a lot.
>
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>> Please share with me if you anyone has an answer or clues to my
>> question regarding the start up performance.
>>
>> Also, one more thing I have observed today is the time taken to run a
>> command on a container went up by more than a second in this latest version.
>>
>> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
>> I call startContainer() to the point the command is started on the
>> container.
>>
>> where as
>>
>> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
>> came to the call back onContainerStarted() to the point the command is seen
>> started running on the container.
>>
>> Thanks,
>> Kishore
>>
>>
>> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have been using the hadoop-2.0.1-beta release candidate and observed
>>> that it is slower in running my simple application that runs on 2
>>> containers. I have tried to find out which parts of it is really having
>>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
>>> found that.
>>>
>>> 1) From the point my Client has submitted the Application Master to RM,
>>> it is taking 2 seconds extra
>>> 2) From the point my container request are set up by Application Master,
>>> till the containers are allocated, it is taking 2 seconds extra
>>>
>>> Is this overhead expected with the changes that went into the new
>>> version? Or is there to improve it by changing something in configurations
>>> or so?
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Omkar,
I have got these number by running a simple C program on the containers
that fetches the timestamp in microseconds and exits. The times mentioned
are low and high, they are not varying so drastically with in a version but
there are huge differences(like a second) between the two versions,
2.0.4-alpha and 2.1.0-beta as I mentioned.
I am using a single node cluster, and all there is absolutely no other
load on the machine/node. My single node cluster is just used for my own
development work, and testing.
I am not aware of what is resource localization, I am not doing anything
specially for that.
Please let me know if you need any other info.
Thanks,
Kishore
On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
>
> More information will help a lot.
>
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>> Please share with me if you anyone has an answer or clues to my
>> question regarding the start up performance.
>>
>> Also, one more thing I have observed today is the time taken to run a
>> command on a container went up by more than a second in this latest version.
>>
>> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
>> I call startContainer() to the point the command is started on the
>> container.
>>
>> where as
>>
>> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
>> came to the call back onContainerStarted() to the point the command is seen
>> started running on the container.
>>
>> Thanks,
>> Kishore
>>
>>
>> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have been using the hadoop-2.0.1-beta release candidate and observed
>>> that it is slower in running my simple application that runs on 2
>>> containers. I have tried to find out which parts of it is really having
>>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
>>> found that.
>>>
>>> 1) From the point my Client has submitted the Application Master to RM,
>>> it is taking 2 seconds extra
>>> 2) From the point my container request are set up by Application Master,
>>> till the containers are allocated, it is taking 2 seconds extra
>>>
>>> Is this overhead expected with the changes that went into the new
>>> version? Or is there to improve it by changing something in configurations
>>> or so?
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Omkar,
I have got these number by running a simple C program on the containers
that fetches the timestamp in microseconds and exits. The times mentioned
are low and high, they are not varying so drastically with in a version but
there are huge differences(like a second) between the two versions,
2.0.4-alpha and 2.1.0-beta as I mentioned.
I am using a single node cluster, and all there is absolutely no other
load on the machine/node. My single node cluster is just used for my own
development work, and testing.
I am not aware of what is resource localization, I am not doing anything
specially for that.
Please let me know if you need any other info.
Thanks,
Kishore
On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
>
> More information will help a lot.
>
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>> Please share with me if you anyone has an answer or clues to my
>> question regarding the start up performance.
>>
>> Also, one more thing I have observed today is the time taken to run a
>> command on a container went up by more than a second in this latest version.
>>
>> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
>> I call startContainer() to the point the command is started on the
>> container.
>>
>> where as
>>
>> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
>> came to the call back onContainerStarted() to the point the command is seen
>> started running on the container.
>>
>> Thanks,
>> Kishore
>>
>>
>> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have been using the hadoop-2.0.1-beta release candidate and observed
>>> that it is slower in running my simple application that runs on 2
>>> containers. I have tried to find out which parts of it is really having
>>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
>>> found that.
>>>
>>> 1) From the point my Client has submitted the Application Master to RM,
>>> it is taking 2 seconds extra
>>> 2) From the point my container request are set up by Application Master,
>>> till the containers are allocated, it is taking 2 seconds extra
>>>
>>> Is this overhead expected with the changes that went into the new
>>> version? Or is there to improve it by changing something in configurations
>>> or so?
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Omkar,
I have got these number by running a simple C program on the containers
that fetches the timestamp in microseconds and exits. The times mentioned
are low and high, they are not varying so drastically with in a version but
there are huge differences(like a second) between the two versions,
2.0.4-alpha and 2.1.0-beta as I mentioned.
I am using a single node cluster, and all there is absolutely no other
load on the machine/node. My single node cluster is just used for my own
development work, and testing.
I am not aware of what is resource localization, I am not doing anything
specially for that.
Please let me know if you need any other info.
Thanks,
Kishore
On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <oj...@hortonworks.com> wrote:
> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
>
> More information will help a lot.
>
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>> Please share with me if you anyone has an answer or clues to my
>> question regarding the start up performance.
>>
>> Also, one more thing I have observed today is the time taken to run a
>> command on a container went up by more than a second in this latest version.
>>
>> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
>> I call startContainer() to the point the command is started on the
>> container.
>>
>> where as
>>
>> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
>> came to the call back onContainerStarted() to the point the command is seen
>> started running on the container.
>>
>> Thanks,
>> Kishore
>>
>>
>> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have been using the hadoop-2.0.1-beta release candidate and observed
>>> that it is slower in running my simple application that runs on 2
>>> containers. I have tried to find out which parts of it is really having
>>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
>>> found that.
>>>
>>> 1) From the point my Client has submitted the Application Master to RM,
>>> it is taking 2 seconds extra
>>> 2) From the point my container request are set up by Application Master,
>>> till the containers are allocated, it is taking 2 seconds extra
>>>
>>> Is this overhead expected with the changes that went into the new
>>> version? Or is there to improve it by changing something in configurations
>>> or so?
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Omkar Joshi <oj...@hortonworks.com>.
How are you making these measurements can you elaborate more? Is it on a
best case basis or on an average or worst case? How many resources are you
sending it for localization? were the sizes and number of these resources
consistent across tests? Were these resources public/private/application
specific? Apart from this is the other load on node manager same? is the
load on hdfs same? did you see any network bottleneck?
More information will help a lot.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi,
> Please share with me if you anyone has an answer or clues to my question
> regarding the start up performance.
>
> Also, one more thing I have observed today is the time taken to run a
> command on a container went up by more than a second in this latest version.
>
> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
> I call startContainer() to the point the command is started on the
> container.
>
> where as
>
> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
> came to the call back onContainerStarted() to the point the command is seen
> started running on the container.
>
> Thanks,
> Kishore
>
>
> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>
>> I have been using the hadoop-2.0.1-beta release candidate and observed
>> that it is slower in running my simple application that runs on 2
>> containers. I have tried to find out which parts of it is really having
>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
>> found that.
>>
>> 1) From the point my Client has submitted the Application Master to RM,
>> it is taking 2 seconds extra
>> 2) From the point my container request are set up by Application Master,
>> till the containers are allocated, it is taking 2 seconds extra
>>
>> Is this overhead expected with the changes that went into the new
>> version? Or is there to improve it by changing something in configurations
>> or so?
>>
>> Thanks,
>> Kishore
>>
>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Omkar Joshi <oj...@hortonworks.com>.
How are you making these measurements can you elaborate more? Is it on a
best case basis or on an average or worst case? How many resources are you
sending it for localization? were the sizes and number of these resources
consistent across tests? Were these resources public/private/application
specific? Apart from this is the other load on node manager same? is the
load on hdfs same? did you see any network bottleneck?
More information will help a lot.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi,
> Please share with me if you anyone has an answer or clues to my question
> regarding the start up performance.
>
> Also, one more thing I have observed today is the time taken to run a
> command on a container went up by more than a second in this latest version.
>
> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
> I call startContainer() to the point the command is started on the
> container.
>
> where as
>
> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
> came to the call back onContainerStarted() to the point the command is seen
> started running on the container.
>
> Thanks,
> Kishore
>
>
> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>
>> I have been using the hadoop-2.0.1-beta release candidate and observed
>> that it is slower in running my simple application that runs on 2
>> containers. I have tried to find out which parts of it is really having
>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
>> found that.
>>
>> 1) From the point my Client has submitted the Application Master to RM,
>> it is taking 2 seconds extra
>> 2) From the point my container request are set up by Application Master,
>> till the containers are allocated, it is taking 2 seconds extra
>>
>> Is this overhead expected with the changes that went into the new
>> version? Or is there to improve it by changing something in configurations
>> or so?
>>
>> Thanks,
>> Kishore
>>
>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Omkar Joshi <oj...@hortonworks.com>.
How are you making these measurements can you elaborate more? Is it on a
best case basis or on an average or worst case? How many resources are you
sending it for localization? were the sizes and number of these resources
consistent across tests? Were these resources public/private/application
specific? Apart from this is the other load on node manager same? is the
load on hdfs same? did you see any network bottleneck?
More information will help a lot.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi,
> Please share with me if you anyone has an answer or clues to my question
> regarding the start up performance.
>
> Also, one more thing I have observed today is the time taken to run a
> command on a container went up by more than a second in this latest version.
>
> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
> I call startContainer() to the point the command is started on the
> container.
>
> where as
>
> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
> came to the call back onContainerStarted() to the point the command is seen
> started running on the container.
>
> Thanks,
> Kishore
>
>
> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>
>> I have been using the hadoop-2.0.1-beta release candidate and observed
>> that it is slower in running my simple application that runs on 2
>> containers. I have tried to find out which parts of it is really having
>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
>> found that.
>>
>> 1) From the point my Client has submitted the Application Master to RM,
>> it is taking 2 seconds extra
>> 2) From the point my container request are set up by Application Master,
>> till the containers are allocated, it is taking 2 seconds extra
>>
>> Is this overhead expected with the changes that went into the new
>> version? Or is there to improve it by changing something in configurations
>> or so?
>>
>> Thanks,
>> Kishore
>>
>
>
Re: Extra start-up overhead with hadoop-2.1.0-beta
Posted by Omkar Joshi <oj...@hortonworks.com>.
How are you making these measurements can you elaborate more? Is it on a
best case basis or on an average or worst case? How many resources are you
sending it for localization? were the sizes and number of these resources
consistent across tests? Were these resources public/private/application
specific? Apart from this is the other load on node manager same? is the
load on hdfs same? did you see any network bottleneck?
More information will help a lot.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi,
> Please share with me if you anyone has an answer or clues to my question
> regarding the start up performance.
>
> Also, one more thing I have observed today is the time taken to run a
> command on a container went up by more than a second in this latest version.
>
> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
> I call startContainer() to the point the command is started on the
> container.
>
> where as
>
> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
> came to the call back onContainerStarted() to the point the command is seen
> started running on the container.
>
> Thanks,
> Kishore
>
>
> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>
>> I have been using the hadoop-2.0.1-beta release candidate and observed
>> that it is slower in running my simple application that runs on 2
>> containers. I have tried to find out which parts of it is really having
>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
>> found that.
>>
>> 1) From the point my Client has submitted the Application Master to RM,
>> it is taking 2 seconds extra
>> 2) From the point my container request are set up by Application Master,
>> till the containers are allocated, it is taking 2 seconds extra
>>
>> Is this overhead expected with the changes that went into the new
>> version? Or is there to improve it by changing something in configurations
>> or so?
>>
>> Thanks,
>> Kishore
>>
>
>