You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hadoop.apache.org by Krishna Kishore Bonagiri <wr...@gmail.com> on 2013/10/21 16:16:11 UTC

Time taken for starting AMRMClientAsync

Hi,
  I am seeing the following call to start() on AMRMClientAsync taking from
0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I
mean does it depend on any of the interval parameters or so in
configuration files? I have tried reducing the value of the first argument
below from 1000 to 100 seconds also, but that doesn't help.

    AMRMClientAsync.CallbackHandler allocListener = new RMCallbackHandler();
    amRMClient = AMRMClientAsync.createAMRMClientAsync(1000, allocListener);
    amRMClient.init(conf);
    amRMClient.start();


Thanks,
Kishore

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  Can you please see if you can answer my question above? I would like to
reduce the time taken by the above calls made by my Application Master, the
way you do.

Thanks,
Kishore


On Tue, Oct 22, 2013 at 3:09 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>   I submit all my applications from a single Client, but all of my
> application masters are taking almost the same amount of time for finishing
> the above calls. Do you reuse ApplicationMaster instances or do some other
> thing for saving this time? Otherwise I felt the fresh application
> connecting to the resource manager would take the same amount of time
> although I don't know why should it take that much?
>
> Thanks,
> Kishore
>
>
> On Mon, Oct 21, 2013 at 9:23 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> Hi Krishna,
>>
>> Those 900ms seems consistent with the numbers we  found while doing some
>> benchmarks in the context of Llama:
>>
>> http://cloudera.github.io/llama/
>>
>> "We found that the first application master created from a client process
>> takes around 900 ms to be ready to submit resource requests. Subsequent
>> application masters created from the same client process take a mean of 20
>> ms. The application master submission throughput (discarding the first
>> submission) tops at approximately 100 application masters per second."
>>
>> I believe there is room for improvement there.
>>
>> Cheers
>>
>>
>> On Mon, Oct 21, 2013 at 7:16 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>> it, I mean does it depend on any of the interval parameters or so in
>>> configuration files? I have tried reducing the value of the first argument
>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>
>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>> RMCallbackHandler();
>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>> allocListener);
>>>     amRMClient.init(conf);
>>>     amRMClient.start();
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>
>>
>> --
>> Alejandro
>>
>
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  Can you please see if you can answer my question above? I would like to
reduce the time taken by the above calls made by my Application Master, the
way you do.

Thanks,
Kishore


On Tue, Oct 22, 2013 at 3:09 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>   I submit all my applications from a single Client, but all of my
> application masters are taking almost the same amount of time for finishing
> the above calls. Do you reuse ApplicationMaster instances or do some other
> thing for saving this time? Otherwise I felt the fresh application
> connecting to the resource manager would take the same amount of time
> although I don't know why should it take that much?
>
> Thanks,
> Kishore
>
>
> On Mon, Oct 21, 2013 at 9:23 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> Hi Krishna,
>>
>> Those 900ms seems consistent with the numbers we  found while doing some
>> benchmarks in the context of Llama:
>>
>> http://cloudera.github.io/llama/
>>
>> "We found that the first application master created from a client process
>> takes around 900 ms to be ready to submit resource requests. Subsequent
>> application masters created from the same client process take a mean of 20
>> ms. The application master submission throughput (discarding the first
>> submission) tops at approximately 100 application masters per second."
>>
>> I believe there is room for improvement there.
>>
>> Cheers
>>
>>
>> On Mon, Oct 21, 2013 at 7:16 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>> it, I mean does it depend on any of the interval parameters or so in
>>> configuration files? I have tried reducing the value of the first argument
>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>
>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>> RMCallbackHandler();
>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>> allocListener);
>>>     amRMClient.init(conf);
>>>     amRMClient.start();
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>
>>
>> --
>> Alejandro
>>
>
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  Can you please see if you can answer my question above? I would like to
reduce the time taken by the above calls made by my Application Master, the
way you do.

Thanks,
Kishore


On Tue, Oct 22, 2013 at 3:09 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>   I submit all my applications from a single Client, but all of my
> application masters are taking almost the same amount of time for finishing
> the above calls. Do you reuse ApplicationMaster instances or do some other
> thing for saving this time? Otherwise I felt the fresh application
> connecting to the resource manager would take the same amount of time
> although I don't know why should it take that much?
>
> Thanks,
> Kishore
>
>
> On Mon, Oct 21, 2013 at 9:23 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> Hi Krishna,
>>
>> Those 900ms seems consistent with the numbers we  found while doing some
>> benchmarks in the context of Llama:
>>
>> http://cloudera.github.io/llama/
>>
>> "We found that the first application master created from a client process
>> takes around 900 ms to be ready to submit resource requests. Subsequent
>> application masters created from the same client process take a mean of 20
>> ms. The application master submission throughput (discarding the first
>> submission) tops at approximately 100 application masters per second."
>>
>> I believe there is room for improvement there.
>>
>> Cheers
>>
>>
>> On Mon, Oct 21, 2013 at 7:16 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>> it, I mean does it depend on any of the interval parameters or so in
>>> configuration files? I have tried reducing the value of the first argument
>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>
>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>> RMCallbackHandler();
>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>> allocListener);
>>>     amRMClient.init(conf);
>>>     amRMClient.start();
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>
>>
>> --
>> Alejandro
>>
>
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  Can you please see if you can answer my question above? I would like to
reduce the time taken by the above calls made by my Application Master, the
way you do.

Thanks,
Kishore


On Tue, Oct 22, 2013 at 3:09 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>   I submit all my applications from a single Client, but all of my
> application masters are taking almost the same amount of time for finishing
> the above calls. Do you reuse ApplicationMaster instances or do some other
> thing for saving this time? Otherwise I felt the fresh application
> connecting to the resource manager would take the same amount of time
> although I don't know why should it take that much?
>
> Thanks,
> Kishore
>
>
> On Mon, Oct 21, 2013 at 9:23 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> Hi Krishna,
>>
>> Those 900ms seems consistent with the numbers we  found while doing some
>> benchmarks in the context of Llama:
>>
>> http://cloudera.github.io/llama/
>>
>> "We found that the first application master created from a client process
>> takes around 900 ms to be ready to submit resource requests. Subsequent
>> application masters created from the same client process take a mean of 20
>> ms. The application master submission throughput (discarding the first
>> submission) tops at approximately 100 application masters per second."
>>
>> I believe there is room for improvement there.
>>
>> Cheers
>>
>>
>> On Mon, Oct 21, 2013 at 7:16 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>> it, I mean does it depend on any of the interval parameters or so in
>>> configuration files? I have tried reducing the value of the first argument
>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>
>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>> RMCallbackHandler();
>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>> allocListener);
>>>     amRMClient.init(conf);
>>>     amRMClient.start();
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>
>>
>> --
>> Alejandro
>>
>
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  I submit all my applications from a single Client, but all of my
application masters are taking almost the same amount of time for finishing
the above calls. Do you reuse ApplicationMaster instances or do some other
thing for saving this time? Otherwise I felt the fresh application
connecting to the resource manager would take the same amount of time
although I don't know why should it take that much?

Thanks,
Kishore


On Mon, Oct 21, 2013 at 9:23 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Hi Krishna,
>
> Those 900ms seems consistent with the numbers we  found while doing some
> benchmarks in the context of Llama:
>
> http://cloudera.github.io/llama/
>
> "We found that the first application master created from a client process
> takes around 900 ms to be ready to submit resource requests. Subsequent
> application masters created from the same client process take a mean of 20
> ms. The application master submission throughput (discarding the first
> submission) tops at approximately 100 application masters per second."
>
> I believe there is room for improvement there.
>
> Cheers
>
>
> On Mon, Oct 21, 2013 at 7:16 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>   I am seeing the following call to start() on AMRMClientAsync taking
>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>> it, I mean does it depend on any of the interval parameters or so in
>> configuration files? I have tried reducing the value of the first argument
>> below from 1000 to 100 seconds also, but that doesn't help.
>>
>>     AMRMClientAsync.CallbackHandler allocListener = new
>> RMCallbackHandler();
>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>> allocListener);
>>     amRMClient.init(conf);
>>     amRMClient.start();
>>
>>
>> Thanks,
>> Kishore
>>
>>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  I submit all my applications from a single Client, but all of my
application masters are taking almost the same amount of time for finishing
the above calls. Do you reuse ApplicationMaster instances or do some other
thing for saving this time? Otherwise I felt the fresh application
connecting to the resource manager would take the same amount of time
although I don't know why should it take that much?

Thanks,
Kishore


On Mon, Oct 21, 2013 at 9:23 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Hi Krishna,
>
> Those 900ms seems consistent with the numbers we  found while doing some
> benchmarks in the context of Llama:
>
> http://cloudera.github.io/llama/
>
> "We found that the first application master created from a client process
> takes around 900 ms to be ready to submit resource requests. Subsequent
> application masters created from the same client process take a mean of 20
> ms. The application master submission throughput (discarding the first
> submission) tops at approximately 100 application masters per second."
>
> I believe there is room for improvement there.
>
> Cheers
>
>
> On Mon, Oct 21, 2013 at 7:16 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>   I am seeing the following call to start() on AMRMClientAsync taking
>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>> it, I mean does it depend on any of the interval parameters or so in
>> configuration files? I have tried reducing the value of the first argument
>> below from 1000 to 100 seconds also, but that doesn't help.
>>
>>     AMRMClientAsync.CallbackHandler allocListener = new
>> RMCallbackHandler();
>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>> allocListener);
>>     amRMClient.init(conf);
>>     amRMClient.start();
>>
>>
>> Thanks,
>> Kishore
>>
>>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  I submit all my applications from a single Client, but all of my
application masters are taking almost the same amount of time for finishing
the above calls. Do you reuse ApplicationMaster instances or do some other
thing for saving this time? Otherwise I felt the fresh application
connecting to the resource manager would take the same amount of time
although I don't know why should it take that much?

Thanks,
Kishore


On Mon, Oct 21, 2013 at 9:23 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Hi Krishna,
>
> Those 900ms seems consistent with the numbers we  found while doing some
> benchmarks in the context of Llama:
>
> http://cloudera.github.io/llama/
>
> "We found that the first application master created from a client process
> takes around 900 ms to be ready to submit resource requests. Subsequent
> application masters created from the same client process take a mean of 20
> ms. The application master submission throughput (discarding the first
> submission) tops at approximately 100 application masters per second."
>
> I believe there is room for improvement there.
>
> Cheers
>
>
> On Mon, Oct 21, 2013 at 7:16 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>   I am seeing the following call to start() on AMRMClientAsync taking
>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>> it, I mean does it depend on any of the interval parameters or so in
>> configuration files? I have tried reducing the value of the first argument
>> below from 1000 to 100 seconds also, but that doesn't help.
>>
>>     AMRMClientAsync.CallbackHandler allocListener = new
>> RMCallbackHandler();
>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>> allocListener);
>>     amRMClient.init(conf);
>>     amRMClient.start();
>>
>>
>> Thanks,
>> Kishore
>>
>>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  I submit all my applications from a single Client, but all of my
application masters are taking almost the same amount of time for finishing
the above calls. Do you reuse ApplicationMaster instances or do some other
thing for saving this time? Otherwise I felt the fresh application
connecting to the resource manager would take the same amount of time
although I don't know why should it take that much?

Thanks,
Kishore


On Mon, Oct 21, 2013 at 9:23 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Hi Krishna,
>
> Those 900ms seems consistent with the numbers we  found while doing some
> benchmarks in the context of Llama:
>
> http://cloudera.github.io/llama/
>
> "We found that the first application master created from a client process
> takes around 900 ms to be ready to submit resource requests. Subsequent
> application masters created from the same client process take a mean of 20
> ms. The application master submission throughput (discarding the first
> submission) tops at approximately 100 application masters per second."
>
> I believe there is room for improvement there.
>
> Cheers
>
>
> On Mon, Oct 21, 2013 at 7:16 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>   I am seeing the following call to start() on AMRMClientAsync taking
>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>> it, I mean does it depend on any of the interval parameters or so in
>> configuration files? I have tried reducing the value of the first argument
>> below from 1000 to 100 seconds also, but that doesn't help.
>>
>>     AMRMClientAsync.CallbackHandler allocListener = new
>> RMCallbackHandler();
>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>> allocListener);
>>     amRMClient.init(conf);
>>     amRMClient.start();
>>
>>
>> Thanks,
>> Kishore
>>
>>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Hi Krishna,

Those 900ms seems consistent with the numbers we  found while doing some
benchmarks in the context of Llama:

http://cloudera.github.io/llama/

"We found that the first application master created from a client process
takes around 900 ms to be ready to submit resource requests. Subsequent
application masters created from the same client process take a mean of 20
ms. The application master submission throughput (discarding the first
submission) tops at approximately 100 application masters per second."

I believe there is room for improvement there.

Cheers

On Mon, Oct 21, 2013 at 7:16 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from
> 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I
> mean does it depend on any of the interval parameters or so in
> configuration files? I have tried reducing the value of the first argument
> below from 1000 to 100 seconds also, but that doesn't help.
>
>     AMRMClientAsync.CallbackHandler allocListener = new
> RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
> allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
>
>
> Thanks,
> Kishore
>
>

-- 
Alejandro

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Vinod,

  Do you expect managed AMs also not take as much time as a second? Or as
Alejandro was saying, only unmanaged AMs?

  I think I am using managed AMs. If managed AMs also are not expected to
take that much time, I shall raise a ticket.

Thanks,
Kishore


On Mon, Nov 18, 2013 at 12:46 AM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

> It is just creating a connection to RM and shouldn't take that long. Can
> you please file a ticket so that we can look at it?
>
> JVM class loading overhead is one possibility but 1 sec is a bit too much.
>
> Thanks,
> +Vinod
>
> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>
> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from
> 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I
> mean does it depend on any of the interval parameters or so in
> configuration files? I have tried reducing the value of the first argument
> below from 1000 to 100 seconds also, but that doesn't help.
>
>     AMRMClientAsync.CallbackHandler allocListener = new
> RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
> allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
>
>
> Thanks,
> Kishore
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Alejandro,

  I am attaching my code for ApplicationMaster and Client for both the
versions of managed and unmanaged AMs here, which are basically copies of
the code that come with Distributed Shell example and modified to be able
to launch multiple AMs from a single Client.

  You can see the calls to printCurrentTimeStamp() that measures time taken
for connecting to the client from AM and also the time taken for starting
up of AM after the Client/UnmangedLauncher submits the application, i.e.
calls submitApplication().

    Here are the times measured using this code. Can you please check what
am I doing different from your code, and suggest what should I change to
reduce these times.

  Time taken for Managed AM Unmanaged AM  AM startup after submission 1.2sec
2.2sec  connecting to RM from AM 0.75sec 0.85sec

  Thanks,
Kishore


On Tue, Nov 26, 2013 at 10:06 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Krishna,
>
> Well, it all depends on your use case. In the case of Llama, Llama is a
> server that hosts multiple unmanaged AMs, thus all AMs run in the same
> process.
>
> Thanks.
>
>
> On Mon, Nov 25, 2013 at 6:40 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Alejandro,
>>
>>   I don't start all the AMs from the same JVM. How can I do that? Also,
>> when I do that, that will save me time taken to get AM started, which is
>> also good to see an improvement in. Please let me know how can I do that?
>> And, would this also save me time taken for connecting from AM to the
>> Resource Manager?
>>
>> Thanks,
>> Kishore
>>
>>
>>
>>
>> On Tue, Nov 26, 2013 at 3:45 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> Hi Krishna,
>>>
>>> Are you starting all AMs from the same JVM? Mind sharing the code you
>>> are using for your time testing?
>>>
>>> Thx
>>>
>>>
>>> On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi Alejandro,
>>>>
>>>>  I have modified the code in
>>>>
>>>>
>>>> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>>>>
>>>> to submit multiple application masters one after another and still
>>>> seeing 800 to 900 ms being taken for the start() call on
>>>> AMRMClientAsync in all of those applications.
>>>>
>>>> Please suggest if you think I am missing something else
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
>>>> write2kishore@gmail.com> wrote:
>>>>
>>>>> Hi Alejandro,
>>>>>
>>>>>   I don't know what are managed and unmanaged AMs, can you please
>>>>> explain me what are the difference and how are each of them launched?
>>>>>
>>>>>  I tried to google for these terms and came
>>>>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>>>>> related to that?
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>>
>>>>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <
>>>>> tucu@cloudera.com> wrote:
>>>>>
>>>>>> Kishore,
>>>>>>
>>>>>> Also, please specify if you are using managed or unmanaged AMs (the
>>>>>> numbers I've mentioned before are using unmanaged AMs).
>>>>>>
>>>>>> thx
>>>>>>
>>>>>>
>>>>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>>>>> vinodkv@hortonworks.com> wrote:
>>>>>>
>>>>>>> It is just creating a connection to RM and shouldn't take that long.
>>>>>>> Can you please file a ticket so that we can look at it?
>>>>>>>
>>>>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>>>>> much.
>>>>>>>
>>>>>>>  Thanks,
>>>>>>> +Vinod
>>>>>>>
>>>>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>   I am seeing the following call to start() on AMRMClientAsync
>>>>>>> taking from 0.9 to 1 second. Why does it take that long? Is there a way to
>>>>>>> reduce it, I mean does it depend on any of the interval parameters or so in
>>>>>>> configuration files? I have tried reducing the value of the first argument
>>>>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>>>>
>>>>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>>>>> RMCallbackHandler();
>>>>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>>>>> allocListener);
>>>>>>>     amRMClient.init(conf);
>>>>>>>     amRMClient.start();
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Kishore
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>> entity to which it is addressed and may contain information that is
>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>> you have received this communication in error, please contact the sender
>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Alejandro
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Alejandro
>>>
>>
>>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Alejandro,

  I am attaching my code for ApplicationMaster and Client for both the
versions of managed and unmanaged AMs here, which are basically copies of
the code that come with Distributed Shell example and modified to be able
to launch multiple AMs from a single Client.

  You can see the calls to printCurrentTimeStamp() that measures time taken
for connecting to the client from AM and also the time taken for starting
up of AM after the Client/UnmangedLauncher submits the application, i.e.
calls submitApplication().

    Here are the times measured using this code. Can you please check what
am I doing different from your code, and suggest what should I change to
reduce these times.

  Time taken for Managed AM Unmanaged AM  AM startup after submission 1.2sec
2.2sec  connecting to RM from AM 0.75sec 0.85sec

  Thanks,
Kishore


On Tue, Nov 26, 2013 at 10:06 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Krishna,
>
> Well, it all depends on your use case. In the case of Llama, Llama is a
> server that hosts multiple unmanaged AMs, thus all AMs run in the same
> process.
>
> Thanks.
>
>
> On Mon, Nov 25, 2013 at 6:40 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Alejandro,
>>
>>   I don't start all the AMs from the same JVM. How can I do that? Also,
>> when I do that, that will save me time taken to get AM started, which is
>> also good to see an improvement in. Please let me know how can I do that?
>> And, would this also save me time taken for connecting from AM to the
>> Resource Manager?
>>
>> Thanks,
>> Kishore
>>
>>
>>
>>
>> On Tue, Nov 26, 2013 at 3:45 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> Hi Krishna,
>>>
>>> Are you starting all AMs from the same JVM? Mind sharing the code you
>>> are using for your time testing?
>>>
>>> Thx
>>>
>>>
>>> On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi Alejandro,
>>>>
>>>>  I have modified the code in
>>>>
>>>>
>>>> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>>>>
>>>> to submit multiple application masters one after another and still
>>>> seeing 800 to 900 ms being taken for the start() call on
>>>> AMRMClientAsync in all of those applications.
>>>>
>>>> Please suggest if you think I am missing something else
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
>>>> write2kishore@gmail.com> wrote:
>>>>
>>>>> Hi Alejandro,
>>>>>
>>>>>   I don't know what are managed and unmanaged AMs, can you please
>>>>> explain me what are the difference and how are each of them launched?
>>>>>
>>>>>  I tried to google for these terms and came
>>>>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>>>>> related to that?
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>>
>>>>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <
>>>>> tucu@cloudera.com> wrote:
>>>>>
>>>>>> Kishore,
>>>>>>
>>>>>> Also, please specify if you are using managed or unmanaged AMs (the
>>>>>> numbers I've mentioned before are using unmanaged AMs).
>>>>>>
>>>>>> thx
>>>>>>
>>>>>>
>>>>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>>>>> vinodkv@hortonworks.com> wrote:
>>>>>>
>>>>>>> It is just creating a connection to RM and shouldn't take that long.
>>>>>>> Can you please file a ticket so that we can look at it?
>>>>>>>
>>>>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>>>>> much.
>>>>>>>
>>>>>>>  Thanks,
>>>>>>> +Vinod
>>>>>>>
>>>>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>   I am seeing the following call to start() on AMRMClientAsync
>>>>>>> taking from 0.9 to 1 second. Why does it take that long? Is there a way to
>>>>>>> reduce it, I mean does it depend on any of the interval parameters or so in
>>>>>>> configuration files? I have tried reducing the value of the first argument
>>>>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>>>>
>>>>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>>>>> RMCallbackHandler();
>>>>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>>>>> allocListener);
>>>>>>>     amRMClient.init(conf);
>>>>>>>     amRMClient.start();
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Kishore
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>> entity to which it is addressed and may contain information that is
>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>> you have received this communication in error, please contact the sender
>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Alejandro
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Alejandro
>>>
>>
>>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Alejandro,

  I am attaching my code for ApplicationMaster and Client for both the
versions of managed and unmanaged AMs here, which are basically copies of
the code that come with Distributed Shell example and modified to be able
to launch multiple AMs from a single Client.

  You can see the calls to printCurrentTimeStamp() that measures time taken
for connecting to the client from AM and also the time taken for starting
up of AM after the Client/UnmangedLauncher submits the application, i.e.
calls submitApplication().

    Here are the times measured using this code. Can you please check what
am I doing different from your code, and suggest what should I change to
reduce these times.

  Time taken for Managed AM Unmanaged AM  AM startup after submission 1.2sec
2.2sec  connecting to RM from AM 0.75sec 0.85sec

  Thanks,
Kishore


On Tue, Nov 26, 2013 at 10:06 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Krishna,
>
> Well, it all depends on your use case. In the case of Llama, Llama is a
> server that hosts multiple unmanaged AMs, thus all AMs run in the same
> process.
>
> Thanks.
>
>
> On Mon, Nov 25, 2013 at 6:40 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Alejandro,
>>
>>   I don't start all the AMs from the same JVM. How can I do that? Also,
>> when I do that, that will save me time taken to get AM started, which is
>> also good to see an improvement in. Please let me know how can I do that?
>> And, would this also save me time taken for connecting from AM to the
>> Resource Manager?
>>
>> Thanks,
>> Kishore
>>
>>
>>
>>
>> On Tue, Nov 26, 2013 at 3:45 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> Hi Krishna,
>>>
>>> Are you starting all AMs from the same JVM? Mind sharing the code you
>>> are using for your time testing?
>>>
>>> Thx
>>>
>>>
>>> On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi Alejandro,
>>>>
>>>>  I have modified the code in
>>>>
>>>>
>>>> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>>>>
>>>> to submit multiple application masters one after another and still
>>>> seeing 800 to 900 ms being taken for the start() call on
>>>> AMRMClientAsync in all of those applications.
>>>>
>>>> Please suggest if you think I am missing something else
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
>>>> write2kishore@gmail.com> wrote:
>>>>
>>>>> Hi Alejandro,
>>>>>
>>>>>   I don't know what are managed and unmanaged AMs, can you please
>>>>> explain me what are the difference and how are each of them launched?
>>>>>
>>>>>  I tried to google for these terms and came
>>>>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>>>>> related to that?
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>>
>>>>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <
>>>>> tucu@cloudera.com> wrote:
>>>>>
>>>>>> Kishore,
>>>>>>
>>>>>> Also, please specify if you are using managed or unmanaged AMs (the
>>>>>> numbers I've mentioned before are using unmanaged AMs).
>>>>>>
>>>>>> thx
>>>>>>
>>>>>>
>>>>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>>>>> vinodkv@hortonworks.com> wrote:
>>>>>>
>>>>>>> It is just creating a connection to RM and shouldn't take that long.
>>>>>>> Can you please file a ticket so that we can look at it?
>>>>>>>
>>>>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>>>>> much.
>>>>>>>
>>>>>>>  Thanks,
>>>>>>> +Vinod
>>>>>>>
>>>>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>   I am seeing the following call to start() on AMRMClientAsync
>>>>>>> taking from 0.9 to 1 second. Why does it take that long? Is there a way to
>>>>>>> reduce it, I mean does it depend on any of the interval parameters or so in
>>>>>>> configuration files? I have tried reducing the value of the first argument
>>>>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>>>>
>>>>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>>>>> RMCallbackHandler();
>>>>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>>>>> allocListener);
>>>>>>>     amRMClient.init(conf);
>>>>>>>     amRMClient.start();
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Kishore
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>> entity to which it is addressed and may contain information that is
>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>> you have received this communication in error, please contact the sender
>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Alejandro
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Alejandro
>>>
>>
>>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Alejandro,

  I am attaching my code for ApplicationMaster and Client for both the
versions of managed and unmanaged AMs here, which are basically copies of
the code that come with Distributed Shell example and modified to be able
to launch multiple AMs from a single Client.

  You can see the calls to printCurrentTimeStamp() that measures time taken
for connecting to the client from AM and also the time taken for starting
up of AM after the Client/UnmangedLauncher submits the application, i.e.
calls submitApplication().

    Here are the times measured using this code. Can you please check what
am I doing different from your code, and suggest what should I change to
reduce these times.

  Time taken for Managed AM Unmanaged AM  AM startup after submission 1.2sec
2.2sec  connecting to RM from AM 0.75sec 0.85sec

  Thanks,
Kishore


On Tue, Nov 26, 2013 at 10:06 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Krishna,
>
> Well, it all depends on your use case. In the case of Llama, Llama is a
> server that hosts multiple unmanaged AMs, thus all AMs run in the same
> process.
>
> Thanks.
>
>
> On Mon, Nov 25, 2013 at 6:40 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Alejandro,
>>
>>   I don't start all the AMs from the same JVM. How can I do that? Also,
>> when I do that, that will save me time taken to get AM started, which is
>> also good to see an improvement in. Please let me know how can I do that?
>> And, would this also save me time taken for connecting from AM to the
>> Resource Manager?
>>
>> Thanks,
>> Kishore
>>
>>
>>
>>
>> On Tue, Nov 26, 2013 at 3:45 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> Hi Krishna,
>>>
>>> Are you starting all AMs from the same JVM? Mind sharing the code you
>>> are using for your time testing?
>>>
>>> Thx
>>>
>>>
>>> On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi Alejandro,
>>>>
>>>>  I have modified the code in
>>>>
>>>>
>>>> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>>>>
>>>> to submit multiple application masters one after another and still
>>>> seeing 800 to 900 ms being taken for the start() call on
>>>> AMRMClientAsync in all of those applications.
>>>>
>>>> Please suggest if you think I am missing something else
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
>>>> write2kishore@gmail.com> wrote:
>>>>
>>>>> Hi Alejandro,
>>>>>
>>>>>   I don't know what are managed and unmanaged AMs, can you please
>>>>> explain me what are the difference and how are each of them launched?
>>>>>
>>>>>  I tried to google for these terms and came
>>>>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>>>>> related to that?
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>>
>>>>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <
>>>>> tucu@cloudera.com> wrote:
>>>>>
>>>>>> Kishore,
>>>>>>
>>>>>> Also, please specify if you are using managed or unmanaged AMs (the
>>>>>> numbers I've mentioned before are using unmanaged AMs).
>>>>>>
>>>>>> thx
>>>>>>
>>>>>>
>>>>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>>>>> vinodkv@hortonworks.com> wrote:
>>>>>>
>>>>>>> It is just creating a connection to RM and shouldn't take that long.
>>>>>>> Can you please file a ticket so that we can look at it?
>>>>>>>
>>>>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>>>>> much.
>>>>>>>
>>>>>>>  Thanks,
>>>>>>> +Vinod
>>>>>>>
>>>>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>   I am seeing the following call to start() on AMRMClientAsync
>>>>>>> taking from 0.9 to 1 second. Why does it take that long? Is there a way to
>>>>>>> reduce it, I mean does it depend on any of the interval parameters or so in
>>>>>>> configuration files? I have tried reducing the value of the first argument
>>>>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>>>>
>>>>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>>>>> RMCallbackHandler();
>>>>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>>>>> allocListener);
>>>>>>>     amRMClient.init(conf);
>>>>>>>     amRMClient.start();
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Kishore
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>> entity to which it is addressed and may contain information that is
>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>> you have received this communication in error, please contact the sender
>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Alejandro
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Alejandro
>>>
>>
>>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Krishna,

Well, it all depends on your use case. In the case of Llama, Llama is a
server that hosts multiple unmanaged AMs, thus all AMs run in the same
process.

Thanks.


On Mon, Nov 25, 2013 at 6:40 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>   I don't start all the AMs from the same JVM. How can I do that? Also,
> when I do that, that will save me time taken to get AM started, which is
> also good to see an improvement in. Please let me know how can I do that?
> And, would this also save me time taken for connecting from AM to the
> Resource Manager?
>
> Thanks,
> Kishore
>
>
>
>
> On Tue, Nov 26, 2013 at 3:45 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> Hi Krishna,
>>
>> Are you starting all AMs from the same JVM? Mind sharing the code you are
>> using for your time testing?
>>
>> Thx
>>
>>
>> On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi Alejandro,
>>>
>>>  I have modified the code in
>>>
>>>
>>> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>>>
>>> to submit multiple application masters one after another and still
>>> seeing 800 to 900 ms being taken for the start() call on
>>> AMRMClientAsync in all of those applications.
>>>
>>> Please suggest if you think I am missing something else
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi Alejandro,
>>>>
>>>>   I don't know what are managed and unmanaged AMs, can you please
>>>> explain me what are the difference and how are each of them launched?
>>>>
>>>>  I tried to google for these terms and came
>>>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>>>> related to that?
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tucu@cloudera.com
>>>> > wrote:
>>>>
>>>>> Kishore,
>>>>>
>>>>> Also, please specify if you are using managed or unmanaged AMs (the
>>>>> numbers I've mentioned before are using unmanaged AMs).
>>>>>
>>>>> thx
>>>>>
>>>>>
>>>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>>>> vinodkv@hortonworks.com> wrote:
>>>>>
>>>>>> It is just creating a connection to RM and shouldn't take that long.
>>>>>> Can you please file a ticket so that we can look at it?
>>>>>>
>>>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>>>> much.
>>>>>>
>>>>>>  Thanks,
>>>>>> +Vinod
>>>>>>
>>>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>>>
>>>>>> Hi,
>>>>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>>>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>>>>> it, I mean does it depend on any of the interval parameters or so in
>>>>>> configuration files? I have tried reducing the value of the first argument
>>>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>>>
>>>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>>>> RMCallbackHandler();
>>>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>>>> allocListener);
>>>>>>     amRMClient.init(conf);
>>>>>>     amRMClient.start();
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Kishore
>>>>>>
>>>>>>
>>>>>>
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to which it is addressed and may contain information that is
>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>> you have received this communication in error, please contact the sender
>>>>>> immediately and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alejandro
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Alejandro
>>
>
>


-- 
Alejandro

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Krishna,

Well, it all depends on your use case. In the case of Llama, Llama is a
server that hosts multiple unmanaged AMs, thus all AMs run in the same
process.

Thanks.


On Mon, Nov 25, 2013 at 6:40 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>   I don't start all the AMs from the same JVM. How can I do that? Also,
> when I do that, that will save me time taken to get AM started, which is
> also good to see an improvement in. Please let me know how can I do that?
> And, would this also save me time taken for connecting from AM to the
> Resource Manager?
>
> Thanks,
> Kishore
>
>
>
>
> On Tue, Nov 26, 2013 at 3:45 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> Hi Krishna,
>>
>> Are you starting all AMs from the same JVM? Mind sharing the code you are
>> using for your time testing?
>>
>> Thx
>>
>>
>> On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi Alejandro,
>>>
>>>  I have modified the code in
>>>
>>>
>>> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>>>
>>> to submit multiple application masters one after another and still
>>> seeing 800 to 900 ms being taken for the start() call on
>>> AMRMClientAsync in all of those applications.
>>>
>>> Please suggest if you think I am missing something else
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi Alejandro,
>>>>
>>>>   I don't know what are managed and unmanaged AMs, can you please
>>>> explain me what are the difference and how are each of them launched?
>>>>
>>>>  I tried to google for these terms and came
>>>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>>>> related to that?
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tucu@cloudera.com
>>>> > wrote:
>>>>
>>>>> Kishore,
>>>>>
>>>>> Also, please specify if you are using managed or unmanaged AMs (the
>>>>> numbers I've mentioned before are using unmanaged AMs).
>>>>>
>>>>> thx
>>>>>
>>>>>
>>>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>>>> vinodkv@hortonworks.com> wrote:
>>>>>
>>>>>> It is just creating a connection to RM and shouldn't take that long.
>>>>>> Can you please file a ticket so that we can look at it?
>>>>>>
>>>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>>>> much.
>>>>>>
>>>>>>  Thanks,
>>>>>> +Vinod
>>>>>>
>>>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>>>
>>>>>> Hi,
>>>>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>>>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>>>>> it, I mean does it depend on any of the interval parameters or so in
>>>>>> configuration files? I have tried reducing the value of the first argument
>>>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>>>
>>>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>>>> RMCallbackHandler();
>>>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>>>> allocListener);
>>>>>>     amRMClient.init(conf);
>>>>>>     amRMClient.start();
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Kishore
>>>>>>
>>>>>>
>>>>>>
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to which it is addressed and may contain information that is
>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>> you have received this communication in error, please contact the sender
>>>>>> immediately and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alejandro
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Alejandro
>>
>
>


-- 
Alejandro

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Krishna,

Well, it all depends on your use case. In the case of Llama, Llama is a
server that hosts multiple unmanaged AMs, thus all AMs run in the same
process.

Thanks.


On Mon, Nov 25, 2013 at 6:40 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>   I don't start all the AMs from the same JVM. How can I do that? Also,
> when I do that, that will save me time taken to get AM started, which is
> also good to see an improvement in. Please let me know how can I do that?
> And, would this also save me time taken for connecting from AM to the
> Resource Manager?
>
> Thanks,
> Kishore
>
>
>
>
> On Tue, Nov 26, 2013 at 3:45 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> Hi Krishna,
>>
>> Are you starting all AMs from the same JVM? Mind sharing the code you are
>> using for your time testing?
>>
>> Thx
>>
>>
>> On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi Alejandro,
>>>
>>>  I have modified the code in
>>>
>>>
>>> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>>>
>>> to submit multiple application masters one after another and still
>>> seeing 800 to 900 ms being taken for the start() call on
>>> AMRMClientAsync in all of those applications.
>>>
>>> Please suggest if you think I am missing something else
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi Alejandro,
>>>>
>>>>   I don't know what are managed and unmanaged AMs, can you please
>>>> explain me what are the difference and how are each of them launched?
>>>>
>>>>  I tried to google for these terms and came
>>>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>>>> related to that?
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tucu@cloudera.com
>>>> > wrote:
>>>>
>>>>> Kishore,
>>>>>
>>>>> Also, please specify if you are using managed or unmanaged AMs (the
>>>>> numbers I've mentioned before are using unmanaged AMs).
>>>>>
>>>>> thx
>>>>>
>>>>>
>>>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>>>> vinodkv@hortonworks.com> wrote:
>>>>>
>>>>>> It is just creating a connection to RM and shouldn't take that long.
>>>>>> Can you please file a ticket so that we can look at it?
>>>>>>
>>>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>>>> much.
>>>>>>
>>>>>>  Thanks,
>>>>>> +Vinod
>>>>>>
>>>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>>>
>>>>>> Hi,
>>>>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>>>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>>>>> it, I mean does it depend on any of the interval parameters or so in
>>>>>> configuration files? I have tried reducing the value of the first argument
>>>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>>>
>>>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>>>> RMCallbackHandler();
>>>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>>>> allocListener);
>>>>>>     amRMClient.init(conf);
>>>>>>     amRMClient.start();
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Kishore
>>>>>>
>>>>>>
>>>>>>
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to which it is addressed and may contain information that is
>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>> you have received this communication in error, please contact the sender
>>>>>> immediately and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alejandro
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Alejandro
>>
>
>


-- 
Alejandro

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Krishna,

Well, it all depends on your use case. In the case of Llama, Llama is a
server that hosts multiple unmanaged AMs, thus all AMs run in the same
process.

Thanks.


On Mon, Nov 25, 2013 at 6:40 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>   I don't start all the AMs from the same JVM. How can I do that? Also,
> when I do that, that will save me time taken to get AM started, which is
> also good to see an improvement in. Please let me know how can I do that?
> And, would this also save me time taken for connecting from AM to the
> Resource Manager?
>
> Thanks,
> Kishore
>
>
>
>
> On Tue, Nov 26, 2013 at 3:45 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> Hi Krishna,
>>
>> Are you starting all AMs from the same JVM? Mind sharing the code you are
>> using for your time testing?
>>
>> Thx
>>
>>
>> On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi Alejandro,
>>>
>>>  I have modified the code in
>>>
>>>
>>> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>>>
>>> to submit multiple application masters one after another and still
>>> seeing 800 to 900 ms being taken for the start() call on
>>> AMRMClientAsync in all of those applications.
>>>
>>> Please suggest if you think I am missing something else
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi Alejandro,
>>>>
>>>>   I don't know what are managed and unmanaged AMs, can you please
>>>> explain me what are the difference and how are each of them launched?
>>>>
>>>>  I tried to google for these terms and came
>>>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>>>> related to that?
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tucu@cloudera.com
>>>> > wrote:
>>>>
>>>>> Kishore,
>>>>>
>>>>> Also, please specify if you are using managed or unmanaged AMs (the
>>>>> numbers I've mentioned before are using unmanaged AMs).
>>>>>
>>>>> thx
>>>>>
>>>>>
>>>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>>>> vinodkv@hortonworks.com> wrote:
>>>>>
>>>>>> It is just creating a connection to RM and shouldn't take that long.
>>>>>> Can you please file a ticket so that we can look at it?
>>>>>>
>>>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>>>> much.
>>>>>>
>>>>>>  Thanks,
>>>>>> +Vinod
>>>>>>
>>>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>>>
>>>>>> Hi,
>>>>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>>>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>>>>> it, I mean does it depend on any of the interval parameters or so in
>>>>>> configuration files? I have tried reducing the value of the first argument
>>>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>>>
>>>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>>>> RMCallbackHandler();
>>>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>>>> allocListener);
>>>>>>     amRMClient.init(conf);
>>>>>>     amRMClient.start();
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Kishore
>>>>>>
>>>>>>
>>>>>>
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to which it is addressed and may contain information that is
>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>> you have received this communication in error, please contact the sender
>>>>>> immediately and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alejandro
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Alejandro
>>
>
>


-- 
Alejandro

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  I don't start all the AMs from the same JVM. How can I do that? Also,
when I do that, that will save me time taken to get AM started, which is
also good to see an improvement in. Please let me know how can I do that?
And, would this also save me time taken for connecting from AM to the
Resource Manager?

Thanks,
Kishore




On Tue, Nov 26, 2013 at 3:45 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Hi Krishna,
>
> Are you starting all AMs from the same JVM? Mind sharing the code you are
> using for your time testing?
>
> Thx
>
>
> On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Alejandro,
>>
>>  I have modified the code in
>>
>>
>> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>>
>> to submit multiple application masters one after another and still seeing
>> 800 to 900 ms being taken for the start() call on AMRMClientAsync in all
>> of those applications.
>>
>> Please suggest if you think I am missing something else
>>
>> Thanks,
>> Kishore
>>
>>
>> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi Alejandro,
>>>
>>>   I don't know what are managed and unmanaged AMs, can you please
>>> explain me what are the difference and how are each of them launched?
>>>
>>>  I tried to google for these terms and came
>>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>>> related to that?
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>>
>>>> Kishore,
>>>>
>>>> Also, please specify if you are using managed or unmanaged AMs (the
>>>> numbers I've mentioned before are using unmanaged AMs).
>>>>
>>>> thx
>>>>
>>>>
>>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>>> vinodkv@hortonworks.com> wrote:
>>>>
>>>>> It is just creating a connection to RM and shouldn't take that long.
>>>>> Can you please file a ticket so that we can look at it?
>>>>>
>>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>>> much.
>>>>>
>>>>>  Thanks,
>>>>> +Vinod
>>>>>
>>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>>
>>>>> Hi,
>>>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>>>> it, I mean does it depend on any of the interval parameters or so in
>>>>> configuration files? I have tried reducing the value of the first argument
>>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>>
>>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>>> RMCallbackHandler();
>>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>>> allocListener);
>>>>>     amRMClient.init(conf);
>>>>>     amRMClient.start();
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>>
>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Alejandro
>>>>
>>>
>>>
>>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  I don't start all the AMs from the same JVM. How can I do that? Also,
when I do that, that will save me time taken to get AM started, which is
also good to see an improvement in. Please let me know how can I do that?
And, would this also save me time taken for connecting from AM to the
Resource Manager?

Thanks,
Kishore




On Tue, Nov 26, 2013 at 3:45 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Hi Krishna,
>
> Are you starting all AMs from the same JVM? Mind sharing the code you are
> using for your time testing?
>
> Thx
>
>
> On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Alejandro,
>>
>>  I have modified the code in
>>
>>
>> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>>
>> to submit multiple application masters one after another and still seeing
>> 800 to 900 ms being taken for the start() call on AMRMClientAsync in all
>> of those applications.
>>
>> Please suggest if you think I am missing something else
>>
>> Thanks,
>> Kishore
>>
>>
>> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi Alejandro,
>>>
>>>   I don't know what are managed and unmanaged AMs, can you please
>>> explain me what are the difference and how are each of them launched?
>>>
>>>  I tried to google for these terms and came
>>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>>> related to that?
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>>
>>>> Kishore,
>>>>
>>>> Also, please specify if you are using managed or unmanaged AMs (the
>>>> numbers I've mentioned before are using unmanaged AMs).
>>>>
>>>> thx
>>>>
>>>>
>>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>>> vinodkv@hortonworks.com> wrote:
>>>>
>>>>> It is just creating a connection to RM and shouldn't take that long.
>>>>> Can you please file a ticket so that we can look at it?
>>>>>
>>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>>> much.
>>>>>
>>>>>  Thanks,
>>>>> +Vinod
>>>>>
>>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>>
>>>>> Hi,
>>>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>>>> it, I mean does it depend on any of the interval parameters or so in
>>>>> configuration files? I have tried reducing the value of the first argument
>>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>>
>>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>>> RMCallbackHandler();
>>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>>> allocListener);
>>>>>     amRMClient.init(conf);
>>>>>     amRMClient.start();
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>>
>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Alejandro
>>>>
>>>
>>>
>>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  I don't start all the AMs from the same JVM. How can I do that? Also,
when I do that, that will save me time taken to get AM started, which is
also good to see an improvement in. Please let me know how can I do that?
And, would this also save me time taken for connecting from AM to the
Resource Manager?

Thanks,
Kishore




On Tue, Nov 26, 2013 at 3:45 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Hi Krishna,
>
> Are you starting all AMs from the same JVM? Mind sharing the code you are
> using for your time testing?
>
> Thx
>
>
> On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Alejandro,
>>
>>  I have modified the code in
>>
>>
>> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>>
>> to submit multiple application masters one after another and still seeing
>> 800 to 900 ms being taken for the start() call on AMRMClientAsync in all
>> of those applications.
>>
>> Please suggest if you think I am missing something else
>>
>> Thanks,
>> Kishore
>>
>>
>> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi Alejandro,
>>>
>>>   I don't know what are managed and unmanaged AMs, can you please
>>> explain me what are the difference and how are each of them launched?
>>>
>>>  I tried to google for these terms and came
>>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>>> related to that?
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>>
>>>> Kishore,
>>>>
>>>> Also, please specify if you are using managed or unmanaged AMs (the
>>>> numbers I've mentioned before are using unmanaged AMs).
>>>>
>>>> thx
>>>>
>>>>
>>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>>> vinodkv@hortonworks.com> wrote:
>>>>
>>>>> It is just creating a connection to RM and shouldn't take that long.
>>>>> Can you please file a ticket so that we can look at it?
>>>>>
>>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>>> much.
>>>>>
>>>>>  Thanks,
>>>>> +Vinod
>>>>>
>>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>>
>>>>> Hi,
>>>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>>>> it, I mean does it depend on any of the interval parameters or so in
>>>>> configuration files? I have tried reducing the value of the first argument
>>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>>
>>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>>> RMCallbackHandler();
>>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>>> allocListener);
>>>>>     amRMClient.init(conf);
>>>>>     amRMClient.start();
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>>
>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Alejandro
>>>>
>>>
>>>
>>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  I don't start all the AMs from the same JVM. How can I do that? Also,
when I do that, that will save me time taken to get AM started, which is
also good to see an improvement in. Please let me know how can I do that?
And, would this also save me time taken for connecting from AM to the
Resource Manager?

Thanks,
Kishore




On Tue, Nov 26, 2013 at 3:45 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Hi Krishna,
>
> Are you starting all AMs from the same JVM? Mind sharing the code you are
> using for your time testing?
>
> Thx
>
>
> On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Alejandro,
>>
>>  I have modified the code in
>>
>>
>> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>>
>> to submit multiple application masters one after another and still seeing
>> 800 to 900 ms being taken for the start() call on AMRMClientAsync in all
>> of those applications.
>>
>> Please suggest if you think I am missing something else
>>
>> Thanks,
>> Kishore
>>
>>
>> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi Alejandro,
>>>
>>>   I don't know what are managed and unmanaged AMs, can you please
>>> explain me what are the difference and how are each of them launched?
>>>
>>>  I tried to google for these terms and came
>>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>>> related to that?
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>>
>>>> Kishore,
>>>>
>>>> Also, please specify if you are using managed or unmanaged AMs (the
>>>> numbers I've mentioned before are using unmanaged AMs).
>>>>
>>>> thx
>>>>
>>>>
>>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>>> vinodkv@hortonworks.com> wrote:
>>>>
>>>>> It is just creating a connection to RM and shouldn't take that long.
>>>>> Can you please file a ticket so that we can look at it?
>>>>>
>>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>>> much.
>>>>>
>>>>>  Thanks,
>>>>> +Vinod
>>>>>
>>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>>
>>>>> Hi,
>>>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>>>> it, I mean does it depend on any of the interval parameters or so in
>>>>> configuration files? I have tried reducing the value of the first argument
>>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>>
>>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>>> RMCallbackHandler();
>>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>>> allocListener);
>>>>>     amRMClient.init(conf);
>>>>>     amRMClient.start();
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>>
>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Alejandro
>>>>
>>>
>>>
>>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Hi Krishna,

Are you starting all AMs from the same JVM? Mind sharing the code you are
using for your time testing?

Thx


On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>  I have modified the code in
>
>
> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>
> to submit multiple application masters one after another and still seeing
> 800 to 900 ms being taken for the start() call on AMRMClientAsync in all
> of those applications.
>
> Please suggest if you think I am missing something else
>
> Thanks,
> Kishore
>
>
> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Alejandro,
>>
>>   I don't know what are managed and unmanaged AMs, can you please explain
>> me what are the difference and how are each of them launched?
>>
>>  I tried to google for these terms and came
>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>> related to that?
>>
>> Thanks,
>> Kishore
>>
>>
>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> Kishore,
>>>
>>> Also, please specify if you are using managed or unmanaged AMs (the
>>> numbers I've mentioned before are using unmanaged AMs).
>>>
>>> thx
>>>
>>>
>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>> vinodkv@hortonworks.com> wrote:
>>>
>>>> It is just creating a connection to RM and shouldn't take that long.
>>>> Can you please file a ticket so that we can look at it?
>>>>
>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>> much.
>>>>
>>>>  Thanks,
>>>> +Vinod
>>>>
>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>
>>>> Hi,
>>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>>> it, I mean does it depend on any of the interval parameters or so in
>>>> configuration files? I have tried reducing the value of the first argument
>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>
>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>> RMCallbackHandler();
>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>> allocListener);
>>>>     amRMClient.init(conf);
>>>>     amRMClient.start();
>>>>
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>>
>>> --
>>> Alejandro
>>>
>>
>>
>


-- 
Alejandro

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Hi Krishna,

Are you starting all AMs from the same JVM? Mind sharing the code you are
using for your time testing?

Thx


On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>  I have modified the code in
>
>
> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>
> to submit multiple application masters one after another and still seeing
> 800 to 900 ms being taken for the start() call on AMRMClientAsync in all
> of those applications.
>
> Please suggest if you think I am missing something else
>
> Thanks,
> Kishore
>
>
> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Alejandro,
>>
>>   I don't know what are managed and unmanaged AMs, can you please explain
>> me what are the difference and how are each of them launched?
>>
>>  I tried to google for these terms and came
>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>> related to that?
>>
>> Thanks,
>> Kishore
>>
>>
>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> Kishore,
>>>
>>> Also, please specify if you are using managed or unmanaged AMs (the
>>> numbers I've mentioned before are using unmanaged AMs).
>>>
>>> thx
>>>
>>>
>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>> vinodkv@hortonworks.com> wrote:
>>>
>>>> It is just creating a connection to RM and shouldn't take that long.
>>>> Can you please file a ticket so that we can look at it?
>>>>
>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>> much.
>>>>
>>>>  Thanks,
>>>> +Vinod
>>>>
>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>
>>>> Hi,
>>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>>> it, I mean does it depend on any of the interval parameters or so in
>>>> configuration files? I have tried reducing the value of the first argument
>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>
>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>> RMCallbackHandler();
>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>> allocListener);
>>>>     amRMClient.init(conf);
>>>>     amRMClient.start();
>>>>
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>>
>>> --
>>> Alejandro
>>>
>>
>>
>


-- 
Alejandro

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Hi Krishna,

Are you starting all AMs from the same JVM? Mind sharing the code you are
using for your time testing?

Thx


On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>  I have modified the code in
>
>
> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>
> to submit multiple application masters one after another and still seeing
> 800 to 900 ms being taken for the start() call on AMRMClientAsync in all
> of those applications.
>
> Please suggest if you think I am missing something else
>
> Thanks,
> Kishore
>
>
> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Alejandro,
>>
>>   I don't know what are managed and unmanaged AMs, can you please explain
>> me what are the difference and how are each of them launched?
>>
>>  I tried to google for these terms and came
>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>> related to that?
>>
>> Thanks,
>> Kishore
>>
>>
>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> Kishore,
>>>
>>> Also, please specify if you are using managed or unmanaged AMs (the
>>> numbers I've mentioned before are using unmanaged AMs).
>>>
>>> thx
>>>
>>>
>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>> vinodkv@hortonworks.com> wrote:
>>>
>>>> It is just creating a connection to RM and shouldn't take that long.
>>>> Can you please file a ticket so that we can look at it?
>>>>
>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>> much.
>>>>
>>>>  Thanks,
>>>> +Vinod
>>>>
>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>
>>>> Hi,
>>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>>> it, I mean does it depend on any of the interval parameters or so in
>>>> configuration files? I have tried reducing the value of the first argument
>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>
>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>> RMCallbackHandler();
>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>> allocListener);
>>>>     amRMClient.init(conf);
>>>>     amRMClient.start();
>>>>
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>>
>>> --
>>> Alejandro
>>>
>>
>>
>


-- 
Alejandro

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Hi Krishna,

Are you starting all AMs from the same JVM? Mind sharing the code you are
using for your time testing?

Thx


On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>  I have modified the code in
>
>
> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
>
> to submit multiple application masters one after another and still seeing
> 800 to 900 ms being taken for the start() call on AMRMClientAsync in all
> of those applications.
>
> Please suggest if you think I am missing something else
>
> Thanks,
> Kishore
>
>
> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Alejandro,
>>
>>   I don't know what are managed and unmanaged AMs, can you please explain
>> me what are the difference and how are each of them launched?
>>
>>  I tried to google for these terms and came
>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
>> related to that?
>>
>> Thanks,
>> Kishore
>>
>>
>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> Kishore,
>>>
>>> Also, please specify if you are using managed or unmanaged AMs (the
>>> numbers I've mentioned before are using unmanaged AMs).
>>>
>>> thx
>>>
>>>
>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>>> vinodkv@hortonworks.com> wrote:
>>>
>>>> It is just creating a connection to RM and shouldn't take that long.
>>>> Can you please file a ticket so that we can look at it?
>>>>
>>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>>> much.
>>>>
>>>>  Thanks,
>>>> +Vinod
>>>>
>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>>
>>>> Hi,
>>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>>> it, I mean does it depend on any of the interval parameters or so in
>>>> configuration files? I have tried reducing the value of the first argument
>>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>>
>>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>>> RMCallbackHandler();
>>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>>> allocListener);
>>>>     amRMClient.init(conf);
>>>>     amRMClient.start();
>>>>
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>>
>>> --
>>> Alejandro
>>>
>>
>>
>


-- 
Alejandro

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

 I have modified the code in

hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java

to submit multiple application masters one after another and still seeing
800 to 900 ms being taken for the start() call on AMRMClientAsync in all of
those applications.

Please suggest if you think I am missing something else

Thanks,
Kishore


On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>   I don't know what are managed and unmanaged AMs, can you please explain
> me what are the difference and how are each of them launched?
>
>  I tried to google for these terms and came
> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
> related to that?
>
> Thanks,
> Kishore
>
>
> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> Kishore,
>>
>> Also, please specify if you are using managed or unmanaged AMs (the
>> numbers I've mentioned before are using unmanaged AMs).
>>
>> thx
>>
>>
>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>> vinodkv@hortonworks.com> wrote:
>>
>>> It is just creating a connection to RM and shouldn't take that long. Can
>>> you please file a ticket so that we can look at it?
>>>
>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>> much.
>>>
>>>  Thanks,
>>> +Vinod
>>>
>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>
>>> Hi,
>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>> it, I mean does it depend on any of the interval parameters or so in
>>> configuration files? I have tried reducing the value of the first argument
>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>
>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>> RMCallbackHandler();
>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>> allocListener);
>>>     amRMClient.init(conf);
>>>     amRMClient.start();
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>>
>> --
>> Alejandro
>>
>
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

 I have modified the code in

hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java

to submit multiple application masters one after another and still seeing
800 to 900 ms being taken for the start() call on AMRMClientAsync in all of
those applications.

Please suggest if you think I am missing something else

Thanks,
Kishore


On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>   I don't know what are managed and unmanaged AMs, can you please explain
> me what are the difference and how are each of them launched?
>
>  I tried to google for these terms and came
> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
> related to that?
>
> Thanks,
> Kishore
>
>
> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> Kishore,
>>
>> Also, please specify if you are using managed or unmanaged AMs (the
>> numbers I've mentioned before are using unmanaged AMs).
>>
>> thx
>>
>>
>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>> vinodkv@hortonworks.com> wrote:
>>
>>> It is just creating a connection to RM and shouldn't take that long. Can
>>> you please file a ticket so that we can look at it?
>>>
>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>> much.
>>>
>>>  Thanks,
>>> +Vinod
>>>
>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>
>>> Hi,
>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>> it, I mean does it depend on any of the interval parameters or so in
>>> configuration files? I have tried reducing the value of the first argument
>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>
>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>> RMCallbackHandler();
>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>> allocListener);
>>>     amRMClient.init(conf);
>>>     amRMClient.start();
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>>
>> --
>> Alejandro
>>
>
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

 I have modified the code in

hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java

to submit multiple application masters one after another and still seeing
800 to 900 ms being taken for the start() call on AMRMClientAsync in all of
those applications.

Please suggest if you think I am missing something else

Thanks,
Kishore


On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>   I don't know what are managed and unmanaged AMs, can you please explain
> me what are the difference and how are each of them launched?
>
>  I tried to google for these terms and came
> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
> related to that?
>
> Thanks,
> Kishore
>
>
> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> Kishore,
>>
>> Also, please specify if you are using managed or unmanaged AMs (the
>> numbers I've mentioned before are using unmanaged AMs).
>>
>> thx
>>
>>
>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>> vinodkv@hortonworks.com> wrote:
>>
>>> It is just creating a connection to RM and shouldn't take that long. Can
>>> you please file a ticket so that we can look at it?
>>>
>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>> much.
>>>
>>>  Thanks,
>>> +Vinod
>>>
>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>
>>> Hi,
>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>> it, I mean does it depend on any of the interval parameters or so in
>>> configuration files? I have tried reducing the value of the first argument
>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>
>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>> RMCallbackHandler();
>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>> allocListener);
>>>     amRMClient.init(conf);
>>>     amRMClient.start();
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>>
>> --
>> Alejandro
>>
>
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

 I have modified the code in

hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java

to submit multiple application masters one after another and still seeing
800 to 900 ms being taken for the start() call on AMRMClientAsync in all of
those applications.

Please suggest if you think I am missing something else

Thanks,
Kishore


On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi Alejandro,
>
>   I don't know what are managed and unmanaged AMs, can you please explain
> me what are the difference and how are each of them launched?
>
>  I tried to google for these terms and came
> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
> related to that?
>
> Thanks,
> Kishore
>
>
> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> Kishore,
>>
>> Also, please specify if you are using managed or unmanaged AMs (the
>> numbers I've mentioned before are using unmanaged AMs).
>>
>> thx
>>
>>
>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
>> vinodkv@hortonworks.com> wrote:
>>
>>> It is just creating a connection to RM and shouldn't take that long. Can
>>> you please file a ticket so that we can look at it?
>>>
>>> JVM class loading overhead is one possibility but 1 sec is a bit too
>>> much.
>>>
>>>  Thanks,
>>> +Vinod
>>>
>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>>
>>> Hi,
>>>   I am seeing the following call to start() on AMRMClientAsync taking
>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>>> it, I mean does it depend on any of the interval parameters or so in
>>> configuration files? I have tried reducing the value of the first argument
>>> below from 1000 to 100 seconds also, but that doesn't help.
>>>
>>>     AMRMClientAsync.CallbackHandler allocListener = new
>>> RMCallbackHandler();
>>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>>> allocListener);
>>>     amRMClient.init(conf);
>>>     amRMClient.start();
>>>
>>>
>>> Thanks,
>>> Kishore
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>>
>> --
>> Alejandro
>>
>
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  I don't know what are managed and unmanaged AMs, can you please explain
me what are the difference and how are each of them launched?

 I tried to google for these terms and came
across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
related to that?

Thanks,
Kishore


On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Kishore,
>
> Also, please specify if you are using managed or unmanaged AMs (the
> numbers I've mentioned before are using unmanaged AMs).
>
> thx
>
>
> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
>> It is just creating a connection to RM and shouldn't take that long. Can
>> you please file a ticket so that we can look at it?
>>
>> JVM class loading overhead is one possibility but 1 sec is a bit too much.
>>
>>  Thanks,
>> +Vinod
>>
>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>
>> Hi,
>>   I am seeing the following call to start() on AMRMClientAsync taking
>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>> it, I mean does it depend on any of the interval parameters or so in
>> configuration files? I have tried reducing the value of the first argument
>> below from 1000 to 100 seconds also, but that doesn't help.
>>
>>     AMRMClientAsync.CallbackHandler allocListener = new
>> RMCallbackHandler();
>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>> allocListener);
>>     amRMClient.init(conf);
>>     amRMClient.start();
>>
>>
>> Thanks,
>> Kishore
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  I don't know what are managed and unmanaged AMs, can you please explain
me what are the difference and how are each of them launched?

 I tried to google for these terms and came
across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
related to that?

Thanks,
Kishore


On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Kishore,
>
> Also, please specify if you are using managed or unmanaged AMs (the
> numbers I've mentioned before are using unmanaged AMs).
>
> thx
>
>
> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
>> It is just creating a connection to RM and shouldn't take that long. Can
>> you please file a ticket so that we can look at it?
>>
>> JVM class loading overhead is one possibility but 1 sec is a bit too much.
>>
>>  Thanks,
>> +Vinod
>>
>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>
>> Hi,
>>   I am seeing the following call to start() on AMRMClientAsync taking
>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>> it, I mean does it depend on any of the interval parameters or so in
>> configuration files? I have tried reducing the value of the first argument
>> below from 1000 to 100 seconds also, but that doesn't help.
>>
>>     AMRMClientAsync.CallbackHandler allocListener = new
>> RMCallbackHandler();
>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>> allocListener);
>>     amRMClient.init(conf);
>>     amRMClient.start();
>>
>>
>> Thanks,
>> Kishore
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  I don't know what are managed and unmanaged AMs, can you please explain
me what are the difference and how are each of them launched?

 I tried to google for these terms and came
across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
related to that?

Thanks,
Kishore


On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Kishore,
>
> Also, please specify if you are using managed or unmanaged AMs (the
> numbers I've mentioned before are using unmanaged AMs).
>
> thx
>
>
> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
>> It is just creating a connection to RM and shouldn't take that long. Can
>> you please file a ticket so that we can look at it?
>>
>> JVM class loading overhead is one possibility but 1 sec is a bit too much.
>>
>>  Thanks,
>> +Vinod
>>
>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>
>> Hi,
>>   I am seeing the following call to start() on AMRMClientAsync taking
>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>> it, I mean does it depend on any of the interval parameters or so in
>> configuration files? I have tried reducing the value of the first argument
>> below from 1000 to 100 seconds also, but that doesn't help.
>>
>>     AMRMClientAsync.CallbackHandler allocListener = new
>> RMCallbackHandler();
>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>> allocListener);
>>     amRMClient.init(conf);
>>     amRMClient.start();
>>
>>
>> Thanks,
>> Kishore
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Hi Alejandro,

  I don't know what are managed and unmanaged AMs, can you please explain
me what are the difference and how are each of them launched?

 I tried to google for these terms and came
across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it
related to that?

Thanks,
Kishore


On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Kishore,
>
> Also, please specify if you are using managed or unmanaged AMs (the
> numbers I've mentioned before are using unmanaged AMs).
>
> thx
>
>
> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
>> It is just creating a connection to RM and shouldn't take that long. Can
>> you please file a ticket so that we can look at it?
>>
>> JVM class loading overhead is one possibility but 1 sec is a bit too much.
>>
>>  Thanks,
>> +Vinod
>>
>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>>
>> Hi,
>>   I am seeing the following call to start() on AMRMClientAsync taking
>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce
>> it, I mean does it depend on any of the interval parameters or so in
>> configuration files? I have tried reducing the value of the first argument
>> below from 1000 to 100 seconds also, but that doesn't help.
>>
>>     AMRMClientAsync.CallbackHandler allocListener = new
>> RMCallbackHandler();
>>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
>> allocListener);
>>     amRMClient.init(conf);
>>     amRMClient.start();
>>
>>
>> Thanks,
>> Kishore
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>
> --
> Alejandro
>

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Kishore,

Also, please specify if you are using managed or unmanaged AMs (the numbers
I've mentioned before are using unmanaged AMs).

thx


On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

> It is just creating a connection to RM and shouldn't take that long. Can
> you please file a ticket so that we can look at it?
>
> JVM class loading overhead is one possibility but 1 sec is a bit too much.
>
>  Thanks,
> +Vinod
>
> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>
> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from
> 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I
> mean does it depend on any of the interval parameters or so in
> configuration files? I have tried reducing the value of the first argument
> below from 1000 to 100 seconds also, but that doesn't help.
>
>     AMRMClientAsync.CallbackHandler allocListener = new
> RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
> allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
>
>
> Thanks,
> Kishore
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.




-- 
Alejandro

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Vinod,

  Do you expect managed AMs also not take as much time as a second? Or as
Alejandro was saying, only unmanaged AMs?

  I think I am using managed AMs. If managed AMs also are not expected to
take that much time, I shall raise a ticket.

Thanks,
Kishore


On Mon, Nov 18, 2013 at 12:46 AM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

> It is just creating a connection to RM and shouldn't take that long. Can
> you please file a ticket so that we can look at it?
>
> JVM class loading overhead is one possibility but 1 sec is a bit too much.
>
> Thanks,
> +Vinod
>
> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>
> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from
> 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I
> mean does it depend on any of the interval parameters or so in
> configuration files? I have tried reducing the value of the first argument
> below from 1000 to 100 seconds also, but that doesn't help.
>
>     AMRMClientAsync.CallbackHandler allocListener = new
> RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
> allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
>
>
> Thanks,
> Kishore
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Vinod,

  Do you expect managed AMs also not take as much time as a second? Or as
Alejandro was saying, only unmanaged AMs?

  I think I am using managed AMs. If managed AMs also are not expected to
take that much time, I shall raise a ticket.

Thanks,
Kishore


On Mon, Nov 18, 2013 at 12:46 AM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

> It is just creating a connection to RM and shouldn't take that long. Can
> you please file a ticket so that we can look at it?
>
> JVM class loading overhead is one possibility but 1 sec is a bit too much.
>
> Thanks,
> +Vinod
>
> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>
> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from
> 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I
> mean does it depend on any of the interval parameters or so in
> configuration files? I have tried reducing the value of the first argument
> below from 1000 to 100 seconds also, but that doesn't help.
>
>     AMRMClientAsync.CallbackHandler allocListener = new
> RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
> allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
>
>
> Thanks,
> Kishore
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: Time taken for starting AMRMClientAsync

Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.

Vinod,

  Do you expect managed AMs also not take as much time as a second? Or as
Alejandro was saying, only unmanaged AMs?

  I think I am using managed AMs. If managed AMs also are not expected to
take that much time, I shall raise a ticket.

Thanks,
Kishore


On Mon, Nov 18, 2013 at 12:46 AM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

> It is just creating a connection to RM and shouldn't take that long. Can
> you please file a ticket so that we can look at it?
>
> JVM class loading overhead is one possibility but 1 sec is a bit too much.
>
> Thanks,
> +Vinod
>
> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>
> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from
> 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I
> mean does it depend on any of the interval parameters or so in
> configuration files? I have tried reducing the value of the first argument
> below from 1000 to 100 seconds also, but that doesn't help.
>
>     AMRMClientAsync.CallbackHandler allocListener = new
> RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
> allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
>
>
> Thanks,
> Kishore
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Kishore,

Also, please specify if you are using managed or unmanaged AMs (the numbers
I've mentioned before are using unmanaged AMs).

thx


On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

> It is just creating a connection to RM and shouldn't take that long. Can
> you please file a ticket so that we can look at it?
>
> JVM class loading overhead is one possibility but 1 sec is a bit too much.
>
>  Thanks,
> +Vinod
>
> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>
> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from
> 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I
> mean does it depend on any of the interval parameters or so in
> configuration files? I have tried reducing the value of the first argument
> below from 1000 to 100 seconds also, but that doesn't help.
>
>     AMRMClientAsync.CallbackHandler allocListener = new
> RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
> allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
>
>
> Thanks,
> Kishore
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.




-- 
Alejandro

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Kishore,

Also, please specify if you are using managed or unmanaged AMs (the numbers
I've mentioned before are using unmanaged AMs).

thx


On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

> It is just creating a connection to RM and shouldn't take that long. Can
> you please file a ticket so that we can look at it?
>
> JVM class loading overhead is one possibility but 1 sec is a bit too much.
>
>  Thanks,
> +Vinod
>
> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>
> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from
> 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I
> mean does it depend on any of the interval parameters or so in
> configuration files? I have tried reducing the value of the first argument
> below from 1000 to 100 seconds also, but that doesn't help.
>
>     AMRMClientAsync.CallbackHandler allocListener = new
> RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
> allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
>
>
> Thanks,
> Kishore
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.




-- 
Alejandro

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Kishore,

Also, please specify if you are using managed or unmanaged AMs (the numbers
I've mentioned before are using unmanaged AMs).

thx


On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

> It is just creating a connection to RM and shouldn't take that long. Can
> you please file a ticket so that we can look at it?
>
> JVM class loading overhead is one possibility but 1 sec is a bit too much.
>
>  Thanks,
> +Vinod
>
> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:
>
> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from
> 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I
> mean does it depend on any of the interval parameters or so in
> configuration files? I have tried reducing the value of the first argument
> below from 1000 to 100 seconds also, but that doesn't help.
>
>     AMRMClientAsync.CallbackHandler allocListener = new
> RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
> allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
>
>
> Thanks,
> Kishore
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.




-- 
Alejandro

Re: Time taken for starting AMRMClientAsync

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.

It is just creating a connection to RM and shouldn't take that long. Can you please file a ticket so that we can look at it?

JVM class loading overhead is one possibility but 1 sec is a bit too much.

Thanks,
+Vinod

On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:

> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I mean does it depend on any of the interval parameters or so in configuration files? I have tried reducing the value of the first argument below from 1000 to 100 seconds also, but that doesn't help.
> 
>     AMRMClientAsync.CallbackHandler allocListener = new RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000, allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
> 
> 
> Thanks,
> Kishore
> 


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Time taken for starting AMRMClientAsync

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.

It is just creating a connection to RM and shouldn't take that long. Can you please file a ticket so that we can look at it?

JVM class loading overhead is one possibility but 1 sec is a bit too much.

Thanks,
+Vinod

On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:

> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I mean does it depend on any of the interval parameters or so in configuration files? I have tried reducing the value of the first argument below from 1000 to 100 seconds also, but that doesn't help.
> 
>     AMRMClientAsync.CallbackHandler allocListener = new RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000, allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
> 
> 
> Thanks,
> Kishore
> 


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Time taken for starting AMRMClientAsync

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.

It is just creating a connection to RM and shouldn't take that long. Can you please file a ticket so that we can look at it?

JVM class loading overhead is one possibility but 1 sec is a bit too much.

Thanks,
+Vinod

On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:

> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I mean does it depend on any of the interval parameters or so in configuration files? I have tried reducing the value of the first argument below from 1000 to 100 seconds also, but that doesn't help.
> 
>     AMRMClientAsync.CallbackHandler allocListener = new RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000, allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
> 
> 
> Thanks,
> Kishore
> 


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Hi Krishna,

Those 900ms seems consistent with the numbers we  found while doing some
benchmarks in the context of Llama:

http://cloudera.github.io/llama/

"We found that the first application master created from a client process
takes around 900 ms to be ready to submit resource requests. Subsequent
application masters created from the same client process take a mean of 20
ms. The application master submission throughput (discarding the first
submission) tops at approximately 100 application masters per second."

I believe there is room for improvement there.

Cheers

On Mon, Oct 21, 2013 at 7:16 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from
> 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I
> mean does it depend on any of the interval parameters or so in
> configuration files? I have tried reducing the value of the first argument
> below from 1000 to 100 seconds also, but that doesn't help.
>
>     AMRMClientAsync.CallbackHandler allocListener = new
> RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
> allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
>
>
> Thanks,
> Kishore
>
>

-- 
Alejandro

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Hi Krishna,

Those 900ms seems consistent with the numbers we  found while doing some
benchmarks in the context of Llama:

http://cloudera.github.io/llama/

"We found that the first application master created from a client process
takes around 900 ms to be ready to submit resource requests. Subsequent
application masters created from the same client process take a mean of 20
ms. The application master submission throughput (discarding the first
submission) tops at approximately 100 application masters per second."

I believe there is room for improvement there.

Cheers

On Mon, Oct 21, 2013 at 7:16 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from
> 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I
> mean does it depend on any of the interval parameters or so in
> configuration files? I have tried reducing the value of the first argument
> below from 1000 to 100 seconds also, but that doesn't help.
>
>     AMRMClientAsync.CallbackHandler allocListener = new
> RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
> allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
>
>
> Thanks,
> Kishore
>
>

-- 
Alejandro

Re: Time taken for starting AMRMClientAsync

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.

It is just creating a connection to RM and shouldn't take that long. Can you please file a ticket so that we can look at it?

JVM class loading overhead is one possibility but 1 sec is a bit too much.

Thanks,
+Vinod

On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:

> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I mean does it depend on any of the interval parameters or so in configuration files? I have tried reducing the value of the first argument below from 1000 to 100 seconds also, but that doesn't help.
> 
>     AMRMClientAsync.CallbackHandler allocListener = new RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000, allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
> 
> 
> Thanks,
> Kishore
> 


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Time taken for starting AMRMClientAsync

Posted by Alejandro Abdelnur <tu...@cloudera.com>.

Hi Krishna,

Those 900ms seems consistent with the numbers we  found while doing some
benchmarks in the context of Llama:

http://cloudera.github.io/llama/

"We found that the first application master created from a client process
takes around 900 ms to be ready to submit resource requests. Subsequent
application masters created from the same client process take a mean of 20
ms. The application master submission throughput (discarding the first
submission) tops at approximately 100 application masters per second."

I believe there is room for improvement there.

Cheers

On Mon, Oct 21, 2013 at 7:16 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:

> Hi,
>   I am seeing the following call to start() on AMRMClientAsync taking from
> 0.9 to 1 second. Why does it take that long? Is there a way to reduce it, I
> mean does it depend on any of the interval parameters or so in
> configuration files? I have tried reducing the value of the first argument
> below from 1000 to 100 seconds also, but that doesn't help.
>
>     AMRMClientAsync.CallbackHandler allocListener = new
> RMCallbackHandler();
>     amRMClient = AMRMClientAsync.createAMRMClientAsync(1000,
> allocListener);
>     amRMClient.init(conf);
>     amRMClient.start();
>
>
> Thanks,
> Kishore
>
>

-- 
Alejandro