You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by sam liu <sa...@gmail.com> on 2014/04/23 05:44:00 UTC
Differences between HistoryServer and Yarn TimeLine server?
Hi Experts,
I am confusing on these two concepts. Could you help explain the
differences?
Thanks!
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
1 and 4:
We have thought about that in addition to service application specific
data, the timeline server should accept the web UI plugin from the
application, install it and render the data on the web page according the
application's design, but still need to figure out the plan. Before that,
the application needs to take care of the data rendering itself, or make
use of third-party monitoring service, such as Ambari, which AFAIK, has
integration with the timeline server in the recent release (Tez is
leveraging it). And yes, it's always welcome if somebody want to contribute.
2:
REST APIs are available for accessing both the generic data and framework
specific data. For the API specification, you can temporally look at the
patch in YARN-1876.
3:
In terms of services, they're almost there. The next step would be about
the security, scalability and integration stuff.
On Thu, Apr 24, 2014 at 11:11 PM, Ashwin Shankar
<as...@gmail.com>wrote:
> Thanks Zhijie !
> I had few more questions :
> 1. I played around with the timeline server ui today which showed the
> generic application history details,
> but I couldn't find any page for application specific data. Is the
> expectation that every application
> needs to build their own UI using the exposed REST apis and somehow
> install it with timeline server ?
> Or am I missing something.
> 2. Are there REST apis for accessing both generic and framework specific
> data in 2.4.0 ?
> 3. Is there an approximate timeframe for timeline server to be feature
> complete ?
> 4. Tez doesn't have any job history UI,is there any work being done to
> integrate Tez with timeline server ?
> If not,is the timeline server ready for such integration in case someone
> wants to pick this up ?
>
> Thanks,
> Ashwin
>
>
>
> On Thu, Apr 24, 2014 at 12:00 AM, Zhijie Shen <zs...@hortonworks.com>wrote:
>
>> Ashwin,
>>
>> YARN-321 focuses on the issue in the scope of generic application history
>> service, while YARN-1530 covers the framework specific data service. And
>> yes, the timeline server is going to cover both.
>>
>> We've not such a Jira before, but it is described in YARN-321's design
>> doc. Anyway, I open a Jira (MAPREDUCE-5858) to track this issue.
>>
>>
>> On Wed, Apr 23, 2014 at 11:25 PM, Ashwin Shankar <
>> ashwinshankar77@gmail.com> wrote:
>>
>>> Hi Zhijie,
>>> There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can
>>> you please let me know what is the
>>> difference ? Is timeline server finally going to be YARN321+YARN1530 ?
>>>
>>> You mentioned that MR is going to integrated with timeline server,is
>>> there a jira I can watch ?
>>>
>>> Thanks,
>>> Ashwin
>>>
>>>
>>> On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>>>
>>>> Sam,
>>>>
>>>> You're right. We can definitely integrate MapReduce to use the timeline
>>>> server to store and serve its specific data, and this is actually our plan.
>>>>
>>>> However, it's a big move, and we still need time to get it done. In
>>>> addition, not to disturb the users that are currently relying on JHS for MR
>>>> job information, we cannot simply remove JHS from Hadoop.
>>>>
>>>>
>>>> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com>wrote:
>>>>
>>>>> Zhijie,
>>>>>
>>>>> I am much clear now. Thanks a lot!
>>>>>
>>>>> As my understanding, besides previous Job History Server, hadoop now
>>>>> has a new timeline server which could restore both the generic YARN
>>>>> application history and the framework specific information. However, I
>>>>> think the timeline server also include the functions of Job History Server,
>>>>> because it can store the framework specific information(of course, include
>>>>> mapreduce framework). In another words, Job History Server is not necessary
>>>>> any more.* If that's the case, why hadoop still include Job History
>>>>> Server?*
>>>>>
>>>>>
>>>>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>>>>
>>>>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>>>>> stage, which actually can serve some generic YARN application history as
>>>>>> well as the framework specific information. Due to the development
>>>>>> logistics, we have created the two concepts: History Server and Timeline
>>>>>> Server. To be simple, you can consider the history server of the service of
>>>>>> the generic YARN application information, while consider the timeline
>>>>>> server of the service of the framework specific information. Importantly,
>>>>>> we just have one daemon, which includes both services, and which we'd like
>>>>>> to call timeline server (unfortunately, the confusing thing is that the
>>>>>> command to start the daemon is "historyserver"). We're going on working on
>>>>>> the timeline server to integrate these two parts, including refactoring the
>>>>>> names.
>>>>>>
>>>>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>>>>> different daemon, which serves the historic information of MapReduce jobs
>>>>>> only.
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com>wrote:
>>>>>>
>>>>>>> Hi Experts,
>>>>>>>
>>>>>>> I am confusing on these two concepts. Could you help explain the
>>>>>>> differences?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Zhijie Shen
>>>>>> Hortonworks Inc.
>>>>>> http://hortonworks.com/
>>>>>>
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to which it is addressed and may contain information that is
>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>> you have received this communication in error, please contact the sender
>>>>>> immediately and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Zhijie Shen
>>>> Hortonworks Inc.
>>>> http://hortonworks.com/
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Ashwin
>>>
>>>
>>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> Thanks,
> Ashwin
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
1 and 4:
We have thought about that in addition to service application specific
data, the timeline server should accept the web UI plugin from the
application, install it and render the data on the web page according the
application's design, but still need to figure out the plan. Before that,
the application needs to take care of the data rendering itself, or make
use of third-party monitoring service, such as Ambari, which AFAIK, has
integration with the timeline server in the recent release (Tez is
leveraging it). And yes, it's always welcome if somebody want to contribute.
2:
REST APIs are available for accessing both the generic data and framework
specific data. For the API specification, you can temporally look at the
patch in YARN-1876.
3:
In terms of services, they're almost there. The next step would be about
the security, scalability and integration stuff.
On Thu, Apr 24, 2014 at 11:11 PM, Ashwin Shankar
<as...@gmail.com>wrote:
> Thanks Zhijie !
> I had few more questions :
> 1. I played around with the timeline server ui today which showed the
> generic application history details,
> but I couldn't find any page for application specific data. Is the
> expectation that every application
> needs to build their own UI using the exposed REST apis and somehow
> install it with timeline server ?
> Or am I missing something.
> 2. Are there REST apis for accessing both generic and framework specific
> data in 2.4.0 ?
> 3. Is there an approximate timeframe for timeline server to be feature
> complete ?
> 4. Tez doesn't have any job history UI,is there any work being done to
> integrate Tez with timeline server ?
> If not,is the timeline server ready for such integration in case someone
> wants to pick this up ?
>
> Thanks,
> Ashwin
>
>
>
> On Thu, Apr 24, 2014 at 12:00 AM, Zhijie Shen <zs...@hortonworks.com>wrote:
>
>> Ashwin,
>>
>> YARN-321 focuses on the issue in the scope of generic application history
>> service, while YARN-1530 covers the framework specific data service. And
>> yes, the timeline server is going to cover both.
>>
>> We've not such a Jira before, but it is described in YARN-321's design
>> doc. Anyway, I open a Jira (MAPREDUCE-5858) to track this issue.
>>
>>
>> On Wed, Apr 23, 2014 at 11:25 PM, Ashwin Shankar <
>> ashwinshankar77@gmail.com> wrote:
>>
>>> Hi Zhijie,
>>> There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can
>>> you please let me know what is the
>>> difference ? Is timeline server finally going to be YARN321+YARN1530 ?
>>>
>>> You mentioned that MR is going to integrated with timeline server,is
>>> there a jira I can watch ?
>>>
>>> Thanks,
>>> Ashwin
>>>
>>>
>>> On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>>>
>>>> Sam,
>>>>
>>>> You're right. We can definitely integrate MapReduce to use the timeline
>>>> server to store and serve its specific data, and this is actually our plan.
>>>>
>>>> However, it's a big move, and we still need time to get it done. In
>>>> addition, not to disturb the users that are currently relying on JHS for MR
>>>> job information, we cannot simply remove JHS from Hadoop.
>>>>
>>>>
>>>> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com>wrote:
>>>>
>>>>> Zhijie,
>>>>>
>>>>> I am much clear now. Thanks a lot!
>>>>>
>>>>> As my understanding, besides previous Job History Server, hadoop now
>>>>> has a new timeline server which could restore both the generic YARN
>>>>> application history and the framework specific information. However, I
>>>>> think the timeline server also include the functions of Job History Server,
>>>>> because it can store the framework specific information(of course, include
>>>>> mapreduce framework). In another words, Job History Server is not necessary
>>>>> any more.* If that's the case, why hadoop still include Job History
>>>>> Server?*
>>>>>
>>>>>
>>>>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>>>>
>>>>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>>>>> stage, which actually can serve some generic YARN application history as
>>>>>> well as the framework specific information. Due to the development
>>>>>> logistics, we have created the two concepts: History Server and Timeline
>>>>>> Server. To be simple, you can consider the history server of the service of
>>>>>> the generic YARN application information, while consider the timeline
>>>>>> server of the service of the framework specific information. Importantly,
>>>>>> we just have one daemon, which includes both services, and which we'd like
>>>>>> to call timeline server (unfortunately, the confusing thing is that the
>>>>>> command to start the daemon is "historyserver"). We're going on working on
>>>>>> the timeline server to integrate these two parts, including refactoring the
>>>>>> names.
>>>>>>
>>>>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>>>>> different daemon, which serves the historic information of MapReduce jobs
>>>>>> only.
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com>wrote:
>>>>>>
>>>>>>> Hi Experts,
>>>>>>>
>>>>>>> I am confusing on these two concepts. Could you help explain the
>>>>>>> differences?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Zhijie Shen
>>>>>> Hortonworks Inc.
>>>>>> http://hortonworks.com/
>>>>>>
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to which it is addressed and may contain information that is
>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>> you have received this communication in error, please contact the sender
>>>>>> immediately and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Zhijie Shen
>>>> Hortonworks Inc.
>>>> http://hortonworks.com/
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Ashwin
>>>
>>>
>>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> Thanks,
> Ashwin
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
1 and 4:
We have thought about that in addition to service application specific
data, the timeline server should accept the web UI plugin from the
application, install it and render the data on the web page according the
application's design, but still need to figure out the plan. Before that,
the application needs to take care of the data rendering itself, or make
use of third-party monitoring service, such as Ambari, which AFAIK, has
integration with the timeline server in the recent release (Tez is
leveraging it). And yes, it's always welcome if somebody want to contribute.
2:
REST APIs are available for accessing both the generic data and framework
specific data. For the API specification, you can temporally look at the
patch in YARN-1876.
3:
In terms of services, they're almost there. The next step would be about
the security, scalability and integration stuff.
On Thu, Apr 24, 2014 at 11:11 PM, Ashwin Shankar
<as...@gmail.com>wrote:
> Thanks Zhijie !
> I had few more questions :
> 1. I played around with the timeline server ui today which showed the
> generic application history details,
> but I couldn't find any page for application specific data. Is the
> expectation that every application
> needs to build their own UI using the exposed REST apis and somehow
> install it with timeline server ?
> Or am I missing something.
> 2. Are there REST apis for accessing both generic and framework specific
> data in 2.4.0 ?
> 3. Is there an approximate timeframe for timeline server to be feature
> complete ?
> 4. Tez doesn't have any job history UI,is there any work being done to
> integrate Tez with timeline server ?
> If not,is the timeline server ready for such integration in case someone
> wants to pick this up ?
>
> Thanks,
> Ashwin
>
>
>
> On Thu, Apr 24, 2014 at 12:00 AM, Zhijie Shen <zs...@hortonworks.com>wrote:
>
>> Ashwin,
>>
>> YARN-321 focuses on the issue in the scope of generic application history
>> service, while YARN-1530 covers the framework specific data service. And
>> yes, the timeline server is going to cover both.
>>
>> We've not such a Jira before, but it is described in YARN-321's design
>> doc. Anyway, I open a Jira (MAPREDUCE-5858) to track this issue.
>>
>>
>> On Wed, Apr 23, 2014 at 11:25 PM, Ashwin Shankar <
>> ashwinshankar77@gmail.com> wrote:
>>
>>> Hi Zhijie,
>>> There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can
>>> you please let me know what is the
>>> difference ? Is timeline server finally going to be YARN321+YARN1530 ?
>>>
>>> You mentioned that MR is going to integrated with timeline server,is
>>> there a jira I can watch ?
>>>
>>> Thanks,
>>> Ashwin
>>>
>>>
>>> On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>>>
>>>> Sam,
>>>>
>>>> You're right. We can definitely integrate MapReduce to use the timeline
>>>> server to store and serve its specific data, and this is actually our plan.
>>>>
>>>> However, it's a big move, and we still need time to get it done. In
>>>> addition, not to disturb the users that are currently relying on JHS for MR
>>>> job information, we cannot simply remove JHS from Hadoop.
>>>>
>>>>
>>>> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com>wrote:
>>>>
>>>>> Zhijie,
>>>>>
>>>>> I am much clear now. Thanks a lot!
>>>>>
>>>>> As my understanding, besides previous Job History Server, hadoop now
>>>>> has a new timeline server which could restore both the generic YARN
>>>>> application history and the framework specific information. However, I
>>>>> think the timeline server also include the functions of Job History Server,
>>>>> because it can store the framework specific information(of course, include
>>>>> mapreduce framework). In another words, Job History Server is not necessary
>>>>> any more.* If that's the case, why hadoop still include Job History
>>>>> Server?*
>>>>>
>>>>>
>>>>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>>>>
>>>>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>>>>> stage, which actually can serve some generic YARN application history as
>>>>>> well as the framework specific information. Due to the development
>>>>>> logistics, we have created the two concepts: History Server and Timeline
>>>>>> Server. To be simple, you can consider the history server of the service of
>>>>>> the generic YARN application information, while consider the timeline
>>>>>> server of the service of the framework specific information. Importantly,
>>>>>> we just have one daemon, which includes both services, and which we'd like
>>>>>> to call timeline server (unfortunately, the confusing thing is that the
>>>>>> command to start the daemon is "historyserver"). We're going on working on
>>>>>> the timeline server to integrate these two parts, including refactoring the
>>>>>> names.
>>>>>>
>>>>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>>>>> different daemon, which serves the historic information of MapReduce jobs
>>>>>> only.
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com>wrote:
>>>>>>
>>>>>>> Hi Experts,
>>>>>>>
>>>>>>> I am confusing on these two concepts. Could you help explain the
>>>>>>> differences?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Zhijie Shen
>>>>>> Hortonworks Inc.
>>>>>> http://hortonworks.com/
>>>>>>
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to which it is addressed and may contain information that is
>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>> you have received this communication in error, please contact the sender
>>>>>> immediately and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Zhijie Shen
>>>> Hortonworks Inc.
>>>> http://hortonworks.com/
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Ashwin
>>>
>>>
>>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> Thanks,
> Ashwin
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
1 and 4:
We have thought about that in addition to service application specific
data, the timeline server should accept the web UI plugin from the
application, install it and render the data on the web page according the
application's design, but still need to figure out the plan. Before that,
the application needs to take care of the data rendering itself, or make
use of third-party monitoring service, such as Ambari, which AFAIK, has
integration with the timeline server in the recent release (Tez is
leveraging it). And yes, it's always welcome if somebody want to contribute.
2:
REST APIs are available for accessing both the generic data and framework
specific data. For the API specification, you can temporally look at the
patch in YARN-1876.
3:
In terms of services, they're almost there. The next step would be about
the security, scalability and integration stuff.
On Thu, Apr 24, 2014 at 11:11 PM, Ashwin Shankar
<as...@gmail.com>wrote:
> Thanks Zhijie !
> I had few more questions :
> 1. I played around with the timeline server ui today which showed the
> generic application history details,
> but I couldn't find any page for application specific data. Is the
> expectation that every application
> needs to build their own UI using the exposed REST apis and somehow
> install it with timeline server ?
> Or am I missing something.
> 2. Are there REST apis for accessing both generic and framework specific
> data in 2.4.0 ?
> 3. Is there an approximate timeframe for timeline server to be feature
> complete ?
> 4. Tez doesn't have any job history UI,is there any work being done to
> integrate Tez with timeline server ?
> If not,is the timeline server ready for such integration in case someone
> wants to pick this up ?
>
> Thanks,
> Ashwin
>
>
>
> On Thu, Apr 24, 2014 at 12:00 AM, Zhijie Shen <zs...@hortonworks.com>wrote:
>
>> Ashwin,
>>
>> YARN-321 focuses on the issue in the scope of generic application history
>> service, while YARN-1530 covers the framework specific data service. And
>> yes, the timeline server is going to cover both.
>>
>> We've not such a Jira before, but it is described in YARN-321's design
>> doc. Anyway, I open a Jira (MAPREDUCE-5858) to track this issue.
>>
>>
>> On Wed, Apr 23, 2014 at 11:25 PM, Ashwin Shankar <
>> ashwinshankar77@gmail.com> wrote:
>>
>>> Hi Zhijie,
>>> There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can
>>> you please let me know what is the
>>> difference ? Is timeline server finally going to be YARN321+YARN1530 ?
>>>
>>> You mentioned that MR is going to integrated with timeline server,is
>>> there a jira I can watch ?
>>>
>>> Thanks,
>>> Ashwin
>>>
>>>
>>> On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>>>
>>>> Sam,
>>>>
>>>> You're right. We can definitely integrate MapReduce to use the timeline
>>>> server to store and serve its specific data, and this is actually our plan.
>>>>
>>>> However, it's a big move, and we still need time to get it done. In
>>>> addition, not to disturb the users that are currently relying on JHS for MR
>>>> job information, we cannot simply remove JHS from Hadoop.
>>>>
>>>>
>>>> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com>wrote:
>>>>
>>>>> Zhijie,
>>>>>
>>>>> I am much clear now. Thanks a lot!
>>>>>
>>>>> As my understanding, besides previous Job History Server, hadoop now
>>>>> has a new timeline server which could restore both the generic YARN
>>>>> application history and the framework specific information. However, I
>>>>> think the timeline server also include the functions of Job History Server,
>>>>> because it can store the framework specific information(of course, include
>>>>> mapreduce framework). In another words, Job History Server is not necessary
>>>>> any more.* If that's the case, why hadoop still include Job History
>>>>> Server?*
>>>>>
>>>>>
>>>>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>>>>
>>>>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>>>>> stage, which actually can serve some generic YARN application history as
>>>>>> well as the framework specific information. Due to the development
>>>>>> logistics, we have created the two concepts: History Server and Timeline
>>>>>> Server. To be simple, you can consider the history server of the service of
>>>>>> the generic YARN application information, while consider the timeline
>>>>>> server of the service of the framework specific information. Importantly,
>>>>>> we just have one daemon, which includes both services, and which we'd like
>>>>>> to call timeline server (unfortunately, the confusing thing is that the
>>>>>> command to start the daemon is "historyserver"). We're going on working on
>>>>>> the timeline server to integrate these two parts, including refactoring the
>>>>>> names.
>>>>>>
>>>>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>>>>> different daemon, which serves the historic information of MapReduce jobs
>>>>>> only.
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com>wrote:
>>>>>>
>>>>>>> Hi Experts,
>>>>>>>
>>>>>>> I am confusing on these two concepts. Could you help explain the
>>>>>>> differences?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Zhijie Shen
>>>>>> Hortonworks Inc.
>>>>>> http://hortonworks.com/
>>>>>>
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to which it is addressed and may contain information that is
>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>> you have received this communication in error, please contact the sender
>>>>>> immediately and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Zhijie Shen
>>>> Hortonworks Inc.
>>>> http://hortonworks.com/
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Ashwin
>>>
>>>
>>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> Thanks,
> Ashwin
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Ashwin Shankar <as...@gmail.com>.
Thanks Zhijie !
I had few more questions :
1. I played around with the timeline server ui today which showed the
generic application history details,
but I couldn't find any page for application specific data. Is the
expectation that every application
needs to build their own UI using the exposed REST apis and somehow install
it with timeline server ?
Or am I missing something.
2. Are there REST apis for accessing both generic and framework specific
data in 2.4.0 ?
3. Is there an approximate timeframe for timeline server to be feature
complete ?
4. Tez doesn't have any job history UI,is there any work being done to
integrate Tez with timeline server ?
If not,is the timeline server ready for such integration in case someone
wants to pick this up ?
Thanks,
Ashwin
On Thu, Apr 24, 2014 at 12:00 AM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Ashwin,
>
> YARN-321 focuses on the issue in the scope of generic application history
> service, while YARN-1530 covers the framework specific data service. And
> yes, the timeline server is going to cover both.
>
> We've not such a Jira before, but it is described in YARN-321's design
> doc. Anyway, I open a Jira (MAPREDUCE-5858) to track this issue.
>
>
> On Wed, Apr 23, 2014 at 11:25 PM, Ashwin Shankar <
> ashwinshankar77@gmail.com> wrote:
>
>> Hi Zhijie,
>> There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can
>> you please let me know what is the
>> difference ? Is timeline server finally going to be YARN321+YARN1530 ?
>>
>> You mentioned that MR is going to integrated with timeline server,is
>> there a jira I can watch ?
>>
>> Thanks,
>> Ashwin
>>
>>
>> On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>>
>>> Sam,
>>>
>>> You're right. We can definitely integrate MapReduce to use the timeline
>>> server to store and serve its specific data, and this is actually our plan.
>>>
>>> However, it's a big move, and we still need time to get it done. In
>>> addition, not to disturb the users that are currently relying on JHS for MR
>>> job information, we cannot simply remove JHS from Hadoop.
>>>
>>>
>>> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
>>>
>>>> Zhijie,
>>>>
>>>> I am much clear now. Thanks a lot!
>>>>
>>>> As my understanding, besides previous Job History Server, hadoop now
>>>> has a new timeline server which could restore both the generic YARN
>>>> application history and the framework specific information. However, I
>>>> think the timeline server also include the functions of Job History Server,
>>>> because it can store the framework specific information(of course, include
>>>> mapreduce framework). In another words, Job History Server is not necessary
>>>> any more.* If that's the case, why hadoop still include Job History
>>>> Server?*
>>>>
>>>>
>>>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>>>
>>>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>>>> stage, which actually can serve some generic YARN application history as
>>>>> well as the framework specific information. Due to the development
>>>>> logistics, we have created the two concepts: History Server and Timeline
>>>>> Server. To be simple, you can consider the history server of the service of
>>>>> the generic YARN application information, while consider the timeline
>>>>> server of the service of the framework specific information. Importantly,
>>>>> we just have one daemon, which includes both services, and which we'd like
>>>>> to call timeline server (unfortunately, the confusing thing is that the
>>>>> command to start the daemon is "historyserver"). We're going on working on
>>>>> the timeline server to integrate these two parts, including refactoring the
>>>>> names.
>>>>>
>>>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>>>> different daemon, which serves the historic information of MapReduce jobs
>>>>> only.
>>>>>
>>>>>
>>>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com>wrote:
>>>>>
>>>>>> Hi Experts,
>>>>>>
>>>>>> I am confusing on these two concepts. Could you help explain the
>>>>>> differences?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Zhijie Shen
>>>>> Hortonworks Inc.
>>>>> http://hortonworks.com/
>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Zhijie Shen
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>>
>>
>>
>>
>> --
>> Thanks,
>> Ashwin
>>
>>
>>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Thanks,
Ashwin
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Ashwin Shankar <as...@gmail.com>.
Thanks Zhijie !
I had few more questions :
1. I played around with the timeline server ui today which showed the
generic application history details,
but I couldn't find any page for application specific data. Is the
expectation that every application
needs to build their own UI using the exposed REST apis and somehow install
it with timeline server ?
Or am I missing something.
2. Are there REST apis for accessing both generic and framework specific
data in 2.4.0 ?
3. Is there an approximate timeframe for timeline server to be feature
complete ?
4. Tez doesn't have any job history UI,is there any work being done to
integrate Tez with timeline server ?
If not,is the timeline server ready for such integration in case someone
wants to pick this up ?
Thanks,
Ashwin
On Thu, Apr 24, 2014 at 12:00 AM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Ashwin,
>
> YARN-321 focuses on the issue in the scope of generic application history
> service, while YARN-1530 covers the framework specific data service. And
> yes, the timeline server is going to cover both.
>
> We've not such a Jira before, but it is described in YARN-321's design
> doc. Anyway, I open a Jira (MAPREDUCE-5858) to track this issue.
>
>
> On Wed, Apr 23, 2014 at 11:25 PM, Ashwin Shankar <
> ashwinshankar77@gmail.com> wrote:
>
>> Hi Zhijie,
>> There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can
>> you please let me know what is the
>> difference ? Is timeline server finally going to be YARN321+YARN1530 ?
>>
>> You mentioned that MR is going to integrated with timeline server,is
>> there a jira I can watch ?
>>
>> Thanks,
>> Ashwin
>>
>>
>> On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>>
>>> Sam,
>>>
>>> You're right. We can definitely integrate MapReduce to use the timeline
>>> server to store and serve its specific data, and this is actually our plan.
>>>
>>> However, it's a big move, and we still need time to get it done. In
>>> addition, not to disturb the users that are currently relying on JHS for MR
>>> job information, we cannot simply remove JHS from Hadoop.
>>>
>>>
>>> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
>>>
>>>> Zhijie,
>>>>
>>>> I am much clear now. Thanks a lot!
>>>>
>>>> As my understanding, besides previous Job History Server, hadoop now
>>>> has a new timeline server which could restore both the generic YARN
>>>> application history and the framework specific information. However, I
>>>> think the timeline server also include the functions of Job History Server,
>>>> because it can store the framework specific information(of course, include
>>>> mapreduce framework). In another words, Job History Server is not necessary
>>>> any more.* If that's the case, why hadoop still include Job History
>>>> Server?*
>>>>
>>>>
>>>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>>>
>>>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>>>> stage, which actually can serve some generic YARN application history as
>>>>> well as the framework specific information. Due to the development
>>>>> logistics, we have created the two concepts: History Server and Timeline
>>>>> Server. To be simple, you can consider the history server of the service of
>>>>> the generic YARN application information, while consider the timeline
>>>>> server of the service of the framework specific information. Importantly,
>>>>> we just have one daemon, which includes both services, and which we'd like
>>>>> to call timeline server (unfortunately, the confusing thing is that the
>>>>> command to start the daemon is "historyserver"). We're going on working on
>>>>> the timeline server to integrate these two parts, including refactoring the
>>>>> names.
>>>>>
>>>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>>>> different daemon, which serves the historic information of MapReduce jobs
>>>>> only.
>>>>>
>>>>>
>>>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com>wrote:
>>>>>
>>>>>> Hi Experts,
>>>>>>
>>>>>> I am confusing on these two concepts. Could you help explain the
>>>>>> differences?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Zhijie Shen
>>>>> Hortonworks Inc.
>>>>> http://hortonworks.com/
>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Zhijie Shen
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>>
>>
>>
>>
>> --
>> Thanks,
>> Ashwin
>>
>>
>>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Thanks,
Ashwin
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Ashwin Shankar <as...@gmail.com>.
Thanks Zhijie !
I had few more questions :
1. I played around with the timeline server ui today which showed the
generic application history details,
but I couldn't find any page for application specific data. Is the
expectation that every application
needs to build their own UI using the exposed REST apis and somehow install
it with timeline server ?
Or am I missing something.
2. Are there REST apis for accessing both generic and framework specific
data in 2.4.0 ?
3. Is there an approximate timeframe for timeline server to be feature
complete ?
4. Tez doesn't have any job history UI,is there any work being done to
integrate Tez with timeline server ?
If not,is the timeline server ready for such integration in case someone
wants to pick this up ?
Thanks,
Ashwin
On Thu, Apr 24, 2014 at 12:00 AM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Ashwin,
>
> YARN-321 focuses on the issue in the scope of generic application history
> service, while YARN-1530 covers the framework specific data service. And
> yes, the timeline server is going to cover both.
>
> We've not such a Jira before, but it is described in YARN-321's design
> doc. Anyway, I open a Jira (MAPREDUCE-5858) to track this issue.
>
>
> On Wed, Apr 23, 2014 at 11:25 PM, Ashwin Shankar <
> ashwinshankar77@gmail.com> wrote:
>
>> Hi Zhijie,
>> There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can
>> you please let me know what is the
>> difference ? Is timeline server finally going to be YARN321+YARN1530 ?
>>
>> You mentioned that MR is going to integrated with timeline server,is
>> there a jira I can watch ?
>>
>> Thanks,
>> Ashwin
>>
>>
>> On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>>
>>> Sam,
>>>
>>> You're right. We can definitely integrate MapReduce to use the timeline
>>> server to store and serve its specific data, and this is actually our plan.
>>>
>>> However, it's a big move, and we still need time to get it done. In
>>> addition, not to disturb the users that are currently relying on JHS for MR
>>> job information, we cannot simply remove JHS from Hadoop.
>>>
>>>
>>> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
>>>
>>>> Zhijie,
>>>>
>>>> I am much clear now. Thanks a lot!
>>>>
>>>> As my understanding, besides previous Job History Server, hadoop now
>>>> has a new timeline server which could restore both the generic YARN
>>>> application history and the framework specific information. However, I
>>>> think the timeline server also include the functions of Job History Server,
>>>> because it can store the framework specific information(of course, include
>>>> mapreduce framework). In another words, Job History Server is not necessary
>>>> any more.* If that's the case, why hadoop still include Job History
>>>> Server?*
>>>>
>>>>
>>>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>>>
>>>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>>>> stage, which actually can serve some generic YARN application history as
>>>>> well as the framework specific information. Due to the development
>>>>> logistics, we have created the two concepts: History Server and Timeline
>>>>> Server. To be simple, you can consider the history server of the service of
>>>>> the generic YARN application information, while consider the timeline
>>>>> server of the service of the framework specific information. Importantly,
>>>>> we just have one daemon, which includes both services, and which we'd like
>>>>> to call timeline server (unfortunately, the confusing thing is that the
>>>>> command to start the daemon is "historyserver"). We're going on working on
>>>>> the timeline server to integrate these two parts, including refactoring the
>>>>> names.
>>>>>
>>>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>>>> different daemon, which serves the historic information of MapReduce jobs
>>>>> only.
>>>>>
>>>>>
>>>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com>wrote:
>>>>>
>>>>>> Hi Experts,
>>>>>>
>>>>>> I am confusing on these two concepts. Could you help explain the
>>>>>> differences?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Zhijie Shen
>>>>> Hortonworks Inc.
>>>>> http://hortonworks.com/
>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Zhijie Shen
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>>
>>
>>
>>
>> --
>> Thanks,
>> Ashwin
>>
>>
>>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Thanks,
Ashwin
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Ashwin Shankar <as...@gmail.com>.
Thanks Zhijie !
I had few more questions :
1. I played around with the timeline server ui today which showed the
generic application history details,
but I couldn't find any page for application specific data. Is the
expectation that every application
needs to build their own UI using the exposed REST apis and somehow install
it with timeline server ?
Or am I missing something.
2. Are there REST apis for accessing both generic and framework specific
data in 2.4.0 ?
3. Is there an approximate timeframe for timeline server to be feature
complete ?
4. Tez doesn't have any job history UI,is there any work being done to
integrate Tez with timeline server ?
If not,is the timeline server ready for such integration in case someone
wants to pick this up ?
Thanks,
Ashwin
On Thu, Apr 24, 2014 at 12:00 AM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Ashwin,
>
> YARN-321 focuses on the issue in the scope of generic application history
> service, while YARN-1530 covers the framework specific data service. And
> yes, the timeline server is going to cover both.
>
> We've not such a Jira before, but it is described in YARN-321's design
> doc. Anyway, I open a Jira (MAPREDUCE-5858) to track this issue.
>
>
> On Wed, Apr 23, 2014 at 11:25 PM, Ashwin Shankar <
> ashwinshankar77@gmail.com> wrote:
>
>> Hi Zhijie,
>> There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can
>> you please let me know what is the
>> difference ? Is timeline server finally going to be YARN321+YARN1530 ?
>>
>> You mentioned that MR is going to integrated with timeline server,is
>> there a jira I can watch ?
>>
>> Thanks,
>> Ashwin
>>
>>
>> On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>>
>>> Sam,
>>>
>>> You're right. We can definitely integrate MapReduce to use the timeline
>>> server to store and serve its specific data, and this is actually our plan.
>>>
>>> However, it's a big move, and we still need time to get it done. In
>>> addition, not to disturb the users that are currently relying on JHS for MR
>>> job information, we cannot simply remove JHS from Hadoop.
>>>
>>>
>>> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
>>>
>>>> Zhijie,
>>>>
>>>> I am much clear now. Thanks a lot!
>>>>
>>>> As my understanding, besides previous Job History Server, hadoop now
>>>> has a new timeline server which could restore both the generic YARN
>>>> application history and the framework specific information. However, I
>>>> think the timeline server also include the functions of Job History Server,
>>>> because it can store the framework specific information(of course, include
>>>> mapreduce framework). In another words, Job History Server is not necessary
>>>> any more.* If that's the case, why hadoop still include Job History
>>>> Server?*
>>>>
>>>>
>>>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>>>
>>>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>>>> stage, which actually can serve some generic YARN application history as
>>>>> well as the framework specific information. Due to the development
>>>>> logistics, we have created the two concepts: History Server and Timeline
>>>>> Server. To be simple, you can consider the history server of the service of
>>>>> the generic YARN application information, while consider the timeline
>>>>> server of the service of the framework specific information. Importantly,
>>>>> we just have one daemon, which includes both services, and which we'd like
>>>>> to call timeline server (unfortunately, the confusing thing is that the
>>>>> command to start the daemon is "historyserver"). We're going on working on
>>>>> the timeline server to integrate these two parts, including refactoring the
>>>>> names.
>>>>>
>>>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>>>> different daemon, which serves the historic information of MapReduce jobs
>>>>> only.
>>>>>
>>>>>
>>>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com>wrote:
>>>>>
>>>>>> Hi Experts,
>>>>>>
>>>>>> I am confusing on these two concepts. Could you help explain the
>>>>>> differences?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Zhijie Shen
>>>>> Hortonworks Inc.
>>>>> http://hortonworks.com/
>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Zhijie Shen
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>>
>>
>>
>>
>> --
>> Thanks,
>> Ashwin
>>
>>
>>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Thanks,
Ashwin
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Ashwin,
YARN-321 focuses on the issue in the scope of generic application history
service, while YARN-1530 covers the framework specific data service. And
yes, the timeline server is going to cover both.
We've not such a Jira before, but it is described in YARN-321's design doc.
Anyway, I open a Jira (MAPREDUCE-5858) to track this issue.
On Wed, Apr 23, 2014 at 11:25 PM, Ashwin Shankar
<as...@gmail.com>wrote:
> Hi Zhijie,
> There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can
> you please let me know what is the
> difference ? Is timeline server finally going to be YARN321+YARN1530 ?
>
> You mentioned that MR is going to integrated with timeline server,is there
> a jira I can watch ?
>
> Thanks,
> Ashwin
>
>
> On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>
>> Sam,
>>
>> You're right. We can definitely integrate MapReduce to use the timeline
>> server to store and serve its specific data, and this is actually our plan.
>>
>> However, it's a big move, and we still need time to get it done. In
>> addition, not to disturb the users that are currently relying on JHS for MR
>> job information, we cannot simply remove JHS from Hadoop.
>>
>>
>> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
>>
>>> Zhijie,
>>>
>>> I am much clear now. Thanks a lot!
>>>
>>> As my understanding, besides previous Job History Server, hadoop now has
>>> a new timeline server which could restore both the generic YARN application
>>> history and the framework specific information. However, I think the
>>> timeline server also include the functions of Job History Server, because
>>> it can store the framework specific information(of course, include
>>> mapreduce framework). In another words, Job History Server is not necessary
>>> any more.* If that's the case, why hadoop still include Job History
>>> Server?*
>>>
>>>
>>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>>
>>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>>> stage, which actually can serve some generic YARN application history as
>>>> well as the framework specific information. Due to the development
>>>> logistics, we have created the two concepts: History Server and Timeline
>>>> Server. To be simple, you can consider the history server of the service of
>>>> the generic YARN application information, while consider the timeline
>>>> server of the service of the framework specific information. Importantly,
>>>> we just have one daemon, which includes both services, and which we'd like
>>>> to call timeline server (unfortunately, the confusing thing is that the
>>>> command to start the daemon is "historyserver"). We're going on working on
>>>> the timeline server to integrate these two parts, including refactoring the
>>>> names.
>>>>
>>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>>> different daemon, which serves the historic information of MapReduce jobs
>>>> only.
>>>>
>>>>
>>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com>wrote:
>>>>
>>>>> Hi Experts,
>>>>>
>>>>> I am confusing on these two concepts. Could you help explain the
>>>>> differences?
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Zhijie Shen
>>>> Hortonworks Inc.
>>>> http://hortonworks.com/
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> Thanks,
> Ashwin
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Ashwin,
YARN-321 focuses on the issue in the scope of generic application history
service, while YARN-1530 covers the framework specific data service. And
yes, the timeline server is going to cover both.
We've not such a Jira before, but it is described in YARN-321's design doc.
Anyway, I open a Jira (MAPREDUCE-5858) to track this issue.
On Wed, Apr 23, 2014 at 11:25 PM, Ashwin Shankar
<as...@gmail.com>wrote:
> Hi Zhijie,
> There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can
> you please let me know what is the
> difference ? Is timeline server finally going to be YARN321+YARN1530 ?
>
> You mentioned that MR is going to integrated with timeline server,is there
> a jira I can watch ?
>
> Thanks,
> Ashwin
>
>
> On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>
>> Sam,
>>
>> You're right. We can definitely integrate MapReduce to use the timeline
>> server to store and serve its specific data, and this is actually our plan.
>>
>> However, it's a big move, and we still need time to get it done. In
>> addition, not to disturb the users that are currently relying on JHS for MR
>> job information, we cannot simply remove JHS from Hadoop.
>>
>>
>> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
>>
>>> Zhijie,
>>>
>>> I am much clear now. Thanks a lot!
>>>
>>> As my understanding, besides previous Job History Server, hadoop now has
>>> a new timeline server which could restore both the generic YARN application
>>> history and the framework specific information. However, I think the
>>> timeline server also include the functions of Job History Server, because
>>> it can store the framework specific information(of course, include
>>> mapreduce framework). In another words, Job History Server is not necessary
>>> any more.* If that's the case, why hadoop still include Job History
>>> Server?*
>>>
>>>
>>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>>
>>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>>> stage, which actually can serve some generic YARN application history as
>>>> well as the framework specific information. Due to the development
>>>> logistics, we have created the two concepts: History Server and Timeline
>>>> Server. To be simple, you can consider the history server of the service of
>>>> the generic YARN application information, while consider the timeline
>>>> server of the service of the framework specific information. Importantly,
>>>> we just have one daemon, which includes both services, and which we'd like
>>>> to call timeline server (unfortunately, the confusing thing is that the
>>>> command to start the daemon is "historyserver"). We're going on working on
>>>> the timeline server to integrate these two parts, including refactoring the
>>>> names.
>>>>
>>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>>> different daemon, which serves the historic information of MapReduce jobs
>>>> only.
>>>>
>>>>
>>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com>wrote:
>>>>
>>>>> Hi Experts,
>>>>>
>>>>> I am confusing on these two concepts. Could you help explain the
>>>>> differences?
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Zhijie Shen
>>>> Hortonworks Inc.
>>>> http://hortonworks.com/
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> Thanks,
> Ashwin
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Ashwin,
YARN-321 focuses on the issue in the scope of generic application history
service, while YARN-1530 covers the framework specific data service. And
yes, the timeline server is going to cover both.
We've not such a Jira before, but it is described in YARN-321's design doc.
Anyway, I open a Jira (MAPREDUCE-5858) to track this issue.
On Wed, Apr 23, 2014 at 11:25 PM, Ashwin Shankar
<as...@gmail.com>wrote:
> Hi Zhijie,
> There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can
> you please let me know what is the
> difference ? Is timeline server finally going to be YARN321+YARN1530 ?
>
> You mentioned that MR is going to integrated with timeline server,is there
> a jira I can watch ?
>
> Thanks,
> Ashwin
>
>
> On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>
>> Sam,
>>
>> You're right. We can definitely integrate MapReduce to use the timeline
>> server to store and serve its specific data, and this is actually our plan.
>>
>> However, it's a big move, and we still need time to get it done. In
>> addition, not to disturb the users that are currently relying on JHS for MR
>> job information, we cannot simply remove JHS from Hadoop.
>>
>>
>> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
>>
>>> Zhijie,
>>>
>>> I am much clear now. Thanks a lot!
>>>
>>> As my understanding, besides previous Job History Server, hadoop now has
>>> a new timeline server which could restore both the generic YARN application
>>> history and the framework specific information. However, I think the
>>> timeline server also include the functions of Job History Server, because
>>> it can store the framework specific information(of course, include
>>> mapreduce framework). In another words, Job History Server is not necessary
>>> any more.* If that's the case, why hadoop still include Job History
>>> Server?*
>>>
>>>
>>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>>
>>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>>> stage, which actually can serve some generic YARN application history as
>>>> well as the framework specific information. Due to the development
>>>> logistics, we have created the two concepts: History Server and Timeline
>>>> Server. To be simple, you can consider the history server of the service of
>>>> the generic YARN application information, while consider the timeline
>>>> server of the service of the framework specific information. Importantly,
>>>> we just have one daemon, which includes both services, and which we'd like
>>>> to call timeline server (unfortunately, the confusing thing is that the
>>>> command to start the daemon is "historyserver"). We're going on working on
>>>> the timeline server to integrate these two parts, including refactoring the
>>>> names.
>>>>
>>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>>> different daemon, which serves the historic information of MapReduce jobs
>>>> only.
>>>>
>>>>
>>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com>wrote:
>>>>
>>>>> Hi Experts,
>>>>>
>>>>> I am confusing on these two concepts. Could you help explain the
>>>>> differences?
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Zhijie Shen
>>>> Hortonworks Inc.
>>>> http://hortonworks.com/
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> Thanks,
> Ashwin
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Ashwin,
YARN-321 focuses on the issue in the scope of generic application history
service, while YARN-1530 covers the framework specific data service. And
yes, the timeline server is going to cover both.
We've not such a Jira before, but it is described in YARN-321's design doc.
Anyway, I open a Jira (MAPREDUCE-5858) to track this issue.
On Wed, Apr 23, 2014 at 11:25 PM, Ashwin Shankar
<as...@gmail.com>wrote:
> Hi Zhijie,
> There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can
> you please let me know what is the
> difference ? Is timeline server finally going to be YARN321+YARN1530 ?
>
> You mentioned that MR is going to integrated with timeline server,is there
> a jira I can watch ?
>
> Thanks,
> Ashwin
>
>
> On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>
>> Sam,
>>
>> You're right. We can definitely integrate MapReduce to use the timeline
>> server to store and serve its specific data, and this is actually our plan.
>>
>> However, it's a big move, and we still need time to get it done. In
>> addition, not to disturb the users that are currently relying on JHS for MR
>> job information, we cannot simply remove JHS from Hadoop.
>>
>>
>> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
>>
>>> Zhijie,
>>>
>>> I am much clear now. Thanks a lot!
>>>
>>> As my understanding, besides previous Job History Server, hadoop now has
>>> a new timeline server which could restore both the generic YARN application
>>> history and the framework specific information. However, I think the
>>> timeline server also include the functions of Job History Server, because
>>> it can store the framework specific information(of course, include
>>> mapreduce framework). In another words, Job History Server is not necessary
>>> any more.* If that's the case, why hadoop still include Job History
>>> Server?*
>>>
>>>
>>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>>
>>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>>> stage, which actually can serve some generic YARN application history as
>>>> well as the framework specific information. Due to the development
>>>> logistics, we have created the two concepts: History Server and Timeline
>>>> Server. To be simple, you can consider the history server of the service of
>>>> the generic YARN application information, while consider the timeline
>>>> server of the service of the framework specific information. Importantly,
>>>> we just have one daemon, which includes both services, and which we'd like
>>>> to call timeline server (unfortunately, the confusing thing is that the
>>>> command to start the daemon is "historyserver"). We're going on working on
>>>> the timeline server to integrate these two parts, including refactoring the
>>>> names.
>>>>
>>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>>> different daemon, which serves the historic information of MapReduce jobs
>>>> only.
>>>>
>>>>
>>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com>wrote:
>>>>
>>>>> Hi Experts,
>>>>>
>>>>> I am confusing on these two concepts. Could you help explain the
>>>>> differences?
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Zhijie Shen
>>>> Hortonworks Inc.
>>>> http://hortonworks.com/
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> Thanks,
> Ashwin
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Ashwin Shankar <as...@gmail.com>.
Hi Zhijie,
There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can you
please let me know what is the
difference ? Is timeline server finally going to be YARN321+YARN1530 ?
You mentioned that MR is going to integrated with timeline server,is there
a jira I can watch ?
Thanks,
Ashwin
On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Sam,
>
> You're right. We can definitely integrate MapReduce to use the timeline
> server to store and serve its specific data, and this is actually our plan.
>
> However, it's a big move, and we still need time to get it done. In
> addition, not to disturb the users that are currently relying on JHS for MR
> job information, we cannot simply remove JHS from Hadoop.
>
>
> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
>
>> Zhijie,
>>
>> I am much clear now. Thanks a lot!
>>
>> As my understanding, besides previous Job History Server, hadoop now has
>> a new timeline server which could restore both the generic YARN application
>> history and the framework specific information. However, I think the
>> timeline server also include the functions of Job History Server, because
>> it can store the framework specific information(of course, include
>> mapreduce framework). In another words, Job History Server is not necessary
>> any more.* If that's the case, why hadoop still include Job History
>> Server?*
>>
>>
>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>
>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>> stage, which actually can serve some generic YARN application history as
>>> well as the framework specific information. Due to the development
>>> logistics, we have created the two concepts: History Server and Timeline
>>> Server. To be simple, you can consider the history server of the service of
>>> the generic YARN application information, while consider the timeline
>>> server of the service of the framework specific information. Importantly,
>>> we just have one daemon, which includes both services, and which we'd like
>>> to call timeline server (unfortunately, the confusing thing is that the
>>> command to start the daemon is "historyserver"). We're going on working on
>>> the timeline server to integrate these two parts, including refactoring the
>>> names.
>>>
>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>> different daemon, which serves the historic information of MapReduce jobs
>>> only.
>>>
>>>
>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
>>>
>>>> Hi Experts,
>>>>
>>>> I am confusing on these two concepts. Could you help explain the
>>>> differences?
>>>>
>>>> Thanks!
>>>>
>>>
>>>
>>>
>>> --
>>> Zhijie Shen
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Thanks,
Ashwin
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Ashwin Shankar <as...@gmail.com>.
Hi Zhijie,
There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can you
please let me know what is the
difference ? Is timeline server finally going to be YARN321+YARN1530 ?
You mentioned that MR is going to integrated with timeline server,is there
a jira I can watch ?
Thanks,
Ashwin
On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Sam,
>
> You're right. We can definitely integrate MapReduce to use the timeline
> server to store and serve its specific data, and this is actually our plan.
>
> However, it's a big move, and we still need time to get it done. In
> addition, not to disturb the users that are currently relying on JHS for MR
> job information, we cannot simply remove JHS from Hadoop.
>
>
> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
>
>> Zhijie,
>>
>> I am much clear now. Thanks a lot!
>>
>> As my understanding, besides previous Job History Server, hadoop now has
>> a new timeline server which could restore both the generic YARN application
>> history and the framework specific information. However, I think the
>> timeline server also include the functions of Job History Server, because
>> it can store the framework specific information(of course, include
>> mapreduce framework). In another words, Job History Server is not necessary
>> any more.* If that's the case, why hadoop still include Job History
>> Server?*
>>
>>
>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>
>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>> stage, which actually can serve some generic YARN application history as
>>> well as the framework specific information. Due to the development
>>> logistics, we have created the two concepts: History Server and Timeline
>>> Server. To be simple, you can consider the history server of the service of
>>> the generic YARN application information, while consider the timeline
>>> server of the service of the framework specific information. Importantly,
>>> we just have one daemon, which includes both services, and which we'd like
>>> to call timeline server (unfortunately, the confusing thing is that the
>>> command to start the daemon is "historyserver"). We're going on working on
>>> the timeline server to integrate these two parts, including refactoring the
>>> names.
>>>
>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>> different daemon, which serves the historic information of MapReduce jobs
>>> only.
>>>
>>>
>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
>>>
>>>> Hi Experts,
>>>>
>>>> I am confusing on these two concepts. Could you help explain the
>>>> differences?
>>>>
>>>> Thanks!
>>>>
>>>
>>>
>>>
>>> --
>>> Zhijie Shen
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Thanks,
Ashwin
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Ashwin Shankar <as...@gmail.com>.
Hi Zhijie,
There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can you
please let me know what is the
difference ? Is timeline server finally going to be YARN321+YARN1530 ?
You mentioned that MR is going to integrated with timeline server,is there
a jira I can watch ?
Thanks,
Ashwin
On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Sam,
>
> You're right. We can definitely integrate MapReduce to use the timeline
> server to store and serve its specific data, and this is actually our plan.
>
> However, it's a big move, and we still need time to get it done. In
> addition, not to disturb the users that are currently relying on JHS for MR
> job information, we cannot simply remove JHS from Hadoop.
>
>
> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
>
>> Zhijie,
>>
>> I am much clear now. Thanks a lot!
>>
>> As my understanding, besides previous Job History Server, hadoop now has
>> a new timeline server which could restore both the generic YARN application
>> history and the framework specific information. However, I think the
>> timeline server also include the functions of Job History Server, because
>> it can store the framework specific information(of course, include
>> mapreduce framework). In another words, Job History Server is not necessary
>> any more.* If that's the case, why hadoop still include Job History
>> Server?*
>>
>>
>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>
>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>> stage, which actually can serve some generic YARN application history as
>>> well as the framework specific information. Due to the development
>>> logistics, we have created the two concepts: History Server and Timeline
>>> Server. To be simple, you can consider the history server of the service of
>>> the generic YARN application information, while consider the timeline
>>> server of the service of the framework specific information. Importantly,
>>> we just have one daemon, which includes both services, and which we'd like
>>> to call timeline server (unfortunately, the confusing thing is that the
>>> command to start the daemon is "historyserver"). We're going on working on
>>> the timeline server to integrate these two parts, including refactoring the
>>> names.
>>>
>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>> different daemon, which serves the historic information of MapReduce jobs
>>> only.
>>>
>>>
>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
>>>
>>>> Hi Experts,
>>>>
>>>> I am confusing on these two concepts. Could you help explain the
>>>> differences?
>>>>
>>>> Thanks!
>>>>
>>>
>>>
>>>
>>> --
>>> Zhijie Shen
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Thanks,
Ashwin
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Ashwin Shankar <as...@gmail.com>.
Hi Zhijie,
There seems to two umbrella jiras for this - YARN-321 and YARN-1530,can you
please let me know what is the
difference ? Is timeline server finally going to be YARN321+YARN1530 ?
You mentioned that MR is going to integrated with timeline server,is there
a jira I can watch ?
Thanks,
Ashwin
On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Sam,
>
> You're right. We can definitely integrate MapReduce to use the timeline
> server to store and serve its specific data, and this is actually our plan.
>
> However, it's a big move, and we still need time to get it done. In
> addition, not to disturb the users that are currently relying on JHS for MR
> job information, we cannot simply remove JHS from Hadoop.
>
>
> On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
>
>> Zhijie,
>>
>> I am much clear now. Thanks a lot!
>>
>> As my understanding, besides previous Job History Server, hadoop now has
>> a new timeline server which could restore both the generic YARN application
>> history and the framework specific information. However, I think the
>> timeline server also include the functions of Job History Server, because
>> it can store the framework specific information(of course, include
>> mapreduce framework). In another words, Job History Server is not necessary
>> any more.* If that's the case, why hadoop still include Job History
>> Server?*
>>
>>
>> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>>
>>> In Hadoop 2.4, we have delivered the timeline server at a preview
>>> stage, which actually can serve some generic YARN application history as
>>> well as the framework specific information. Due to the development
>>> logistics, we have created the two concepts: History Server and Timeline
>>> Server. To be simple, you can consider the history server of the service of
>>> the generic YARN application information, while consider the timeline
>>> server of the service of the framework specific information. Importantly,
>>> we just have one daemon, which includes both services, and which we'd like
>>> to call timeline server (unfortunately, the confusing thing is that the
>>> command to start the daemon is "historyserver"). We're going on working on
>>> the timeline server to integrate these two parts, including refactoring the
>>> names.
>>>
>>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>>> different daemon, which serves the historic information of MapReduce jobs
>>> only.
>>>
>>>
>>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
>>>
>>>> Hi Experts,
>>>>
>>>> I am confusing on these two concepts. Could you help explain the
>>>> differences?
>>>>
>>>> Thanks!
>>>>
>>>
>>>
>>>
>>> --
>>> Zhijie Shen
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Thanks,
Ashwin
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Sam,
You're right. We can definitely integrate MapReduce to use the timeline
server to store and serve its specific data, and this is actually our plan.
However, it's a big move, and we still need time to get it done. In
addition, not to disturb the users that are currently relying on JHS for MR
job information, we cannot simply remove JHS from Hadoop.
On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
> Zhijie,
>
> I am much clear now. Thanks a lot!
>
> As my understanding, besides previous Job History Server, hadoop now has a
> new timeline server which could restore both the generic YARN application
> history and the framework specific information. However, I think the
> timeline server also include the functions of Job History Server, because
> it can store the framework specific information(of course, include
> mapreduce framework). In another words, Job History Server is not necessary
> any more.* If that's the case, why hadoop still include Job History
> Server?*
>
>
> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>
>> In Hadoop 2.4, we have delivered the timeline server at a preview stage,
>> which actually can serve some generic YARN application history as well as
>> the framework specific information. Due to the development logistics, we
>> have created the two concepts: History Server and Timeline Server. To be
>> simple, you can consider the history server of the service of the generic
>> YARN application information, while consider the timeline server of the
>> service of the framework specific information. Importantly, we just have
>> one daemon, which includes both services, and which we'd like to call
>> timeline server (unfortunately, the confusing thing is that the command to
>> start the daemon is "historyserver"). We're going on working on the
>> timeline server to integrate these two parts, including refactoring the
>> names.
>>
>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>> different daemon, which serves the historic information of MapReduce jobs
>> only.
>>
>>
>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
>>
>>> Hi Experts,
>>>
>>> I am confusing on these two concepts. Could you help explain the
>>> differences?
>>>
>>> Thanks!
>>>
>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Sam,
You're right. We can definitely integrate MapReduce to use the timeline
server to store and serve its specific data, and this is actually our plan.
However, it's a big move, and we still need time to get it done. In
addition, not to disturb the users that are currently relying on JHS for MR
job information, we cannot simply remove JHS from Hadoop.
On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
> Zhijie,
>
> I am much clear now. Thanks a lot!
>
> As my understanding, besides previous Job History Server, hadoop now has a
> new timeline server which could restore both the generic YARN application
> history and the framework specific information. However, I think the
> timeline server also include the functions of Job History Server, because
> it can store the framework specific information(of course, include
> mapreduce framework). In another words, Job History Server is not necessary
> any more.* If that's the case, why hadoop still include Job History
> Server?*
>
>
> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>
>> In Hadoop 2.4, we have delivered the timeline server at a preview stage,
>> which actually can serve some generic YARN application history as well as
>> the framework specific information. Due to the development logistics, we
>> have created the two concepts: History Server and Timeline Server. To be
>> simple, you can consider the history server of the service of the generic
>> YARN application information, while consider the timeline server of the
>> service of the framework specific information. Importantly, we just have
>> one daemon, which includes both services, and which we'd like to call
>> timeline server (unfortunately, the confusing thing is that the command to
>> start the daemon is "historyserver"). We're going on working on the
>> timeline server to integrate these two parts, including refactoring the
>> names.
>>
>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>> different daemon, which serves the historic information of MapReduce jobs
>> only.
>>
>>
>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
>>
>>> Hi Experts,
>>>
>>> I am confusing on these two concepts. Could you help explain the
>>> differences?
>>>
>>> Thanks!
>>>
>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Sam,
You're right. We can definitely integrate MapReduce to use the timeline
server to store and serve its specific data, and this is actually our plan.
However, it's a big move, and we still need time to get it done. In
addition, not to disturb the users that are currently relying on JHS for MR
job information, we cannot simply remove JHS from Hadoop.
On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
> Zhijie,
>
> I am much clear now. Thanks a lot!
>
> As my understanding, besides previous Job History Server, hadoop now has a
> new timeline server which could restore both the generic YARN application
> history and the framework specific information. However, I think the
> timeline server also include the functions of Job History Server, because
> it can store the framework specific information(of course, include
> mapreduce framework). In another words, Job History Server is not necessary
> any more.* If that's the case, why hadoop still include Job History
> Server?*
>
>
> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>
>> In Hadoop 2.4, we have delivered the timeline server at a preview stage,
>> which actually can serve some generic YARN application history as well as
>> the framework specific information. Due to the development logistics, we
>> have created the two concepts: History Server and Timeline Server. To be
>> simple, you can consider the history server of the service of the generic
>> YARN application information, while consider the timeline server of the
>> service of the framework specific information. Importantly, we just have
>> one daemon, which includes both services, and which we'd like to call
>> timeline server (unfortunately, the confusing thing is that the command to
>> start the daemon is "historyserver"). We're going on working on the
>> timeline server to integrate these two parts, including refactoring the
>> names.
>>
>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>> different daemon, which serves the historic information of MapReduce jobs
>> only.
>>
>>
>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
>>
>>> Hi Experts,
>>>
>>> I am confusing on these two concepts. Could you help explain the
>>> differences?
>>>
>>> Thanks!
>>>
>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Sam,
You're right. We can definitely integrate MapReduce to use the timeline
server to store and serve its specific data, and this is actually our plan.
However, it's a big move, and we still need time to get it done. In
addition, not to disturb the users that are currently relying on JHS for MR
job information, we cannot simply remove JHS from Hadoop.
On Wed, Apr 23, 2014 at 8:15 PM, sam liu <sa...@gmail.com> wrote:
> Zhijie,
>
> I am much clear now. Thanks a lot!
>
> As my understanding, besides previous Job History Server, hadoop now has a
> new timeline server which could restore both the generic YARN application
> history and the framework specific information. However, I think the
> timeline server also include the functions of Job History Server, because
> it can store the framework specific information(of course, include
> mapreduce framework). In another words, Job History Server is not necessary
> any more.* If that's the case, why hadoop still include Job History
> Server?*
>
>
> 2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
>
>> In Hadoop 2.4, we have delivered the timeline server at a preview stage,
>> which actually can serve some generic YARN application history as well as
>> the framework specific information. Due to the development logistics, we
>> have created the two concepts: History Server and Timeline Server. To be
>> simple, you can consider the history server of the service of the generic
>> YARN application information, while consider the timeline server of the
>> service of the framework specific information. Importantly, we just have
>> one daemon, which includes both services, and which we'd like to call
>> timeline server (unfortunately, the confusing thing is that the command to
>> start the daemon is "historyserver"). We're going on working on the
>> timeline server to integrate these two parts, including refactoring the
>> names.
>>
>> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
>> different daemon, which serves the historic information of MapReduce jobs
>> only.
>>
>>
>> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
>>
>>> Hi Experts,
>>>
>>> I am confusing on these two concepts. Could you help explain the
>>> differences?
>>>
>>> Thanks!
>>>
>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by sam liu <sa...@gmail.com>.
Zhijie,
I am much clear now. Thanks a lot!
As my understanding, besides previous Job History Server, hadoop now has a
new timeline server which could restore both the generic YARN application
history and the framework specific information. However, I think the
timeline server also include the functions of Job History Server, because
it can store the framework specific information(of course, include
mapreduce framework). In another words, Job History Server is not necessary
any more.* If that's the case, why hadoop still include Job History Server?*
2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
> In Hadoop 2.4, we have delivered the timeline server at a preview stage,
> which actually can serve some generic YARN application history as well as
> the framework specific information. Due to the development logistics, we
> have created the two concepts: History Server and Timeline Server. To be
> simple, you can consider the history server of the service of the generic
> YARN application information, while consider the timeline server of the
> service of the framework specific information. Importantly, we just have
> one daemon, which includes both services, and which we'd like to call
> timeline server (unfortunately, the confusing thing is that the command to
> start the daemon is "historyserver"). We're going on working on the
> timeline server to integrate these two parts, including refactoring the
> names.
>
> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
> different daemon, which serves the historic information of MapReduce jobs
> only.
>
>
> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
>
>> Hi Experts,
>>
>> I am confusing on these two concepts. Could you help explain the
>> differences?
>>
>> Thanks!
>>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by sam liu <sa...@gmail.com>.
Zhijie,
I am much clear now. Thanks a lot!
As my understanding, besides previous Job History Server, hadoop now has a
new timeline server which could restore both the generic YARN application
history and the framework specific information. However, I think the
timeline server also include the functions of Job History Server, because
it can store the framework specific information(of course, include
mapreduce framework). In another words, Job History Server is not necessary
any more.* If that's the case, why hadoop still include Job History Server?*
2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
> In Hadoop 2.4, we have delivered the timeline server at a preview stage,
> which actually can serve some generic YARN application history as well as
> the framework specific information. Due to the development logistics, we
> have created the two concepts: History Server and Timeline Server. To be
> simple, you can consider the history server of the service of the generic
> YARN application information, while consider the timeline server of the
> service of the framework specific information. Importantly, we just have
> one daemon, which includes both services, and which we'd like to call
> timeline server (unfortunately, the confusing thing is that the command to
> start the daemon is "historyserver"). We're going on working on the
> timeline server to integrate these two parts, including refactoring the
> names.
>
> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
> different daemon, which serves the historic information of MapReduce jobs
> only.
>
>
> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
>
>> Hi Experts,
>>
>> I am confusing on these two concepts. Could you help explain the
>> differences?
>>
>> Thanks!
>>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by sam liu <sa...@gmail.com>.
Zhijie,
I am much clear now. Thanks a lot!
As my understanding, besides previous Job History Server, hadoop now has a
new timeline server which could restore both the generic YARN application
history and the framework specific information. However, I think the
timeline server also include the functions of Job History Server, because
it can store the framework specific information(of course, include
mapreduce framework). In another words, Job History Server is not necessary
any more.* If that's the case, why hadoop still include Job History Server?*
2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
> In Hadoop 2.4, we have delivered the timeline server at a preview stage,
> which actually can serve some generic YARN application history as well as
> the framework specific information. Due to the development logistics, we
> have created the two concepts: History Server and Timeline Server. To be
> simple, you can consider the history server of the service of the generic
> YARN application information, while consider the timeline server of the
> service of the framework specific information. Importantly, we just have
> one daemon, which includes both services, and which we'd like to call
> timeline server (unfortunately, the confusing thing is that the command to
> start the daemon is "historyserver"). We're going on working on the
> timeline server to integrate these two parts, including refactoring the
> names.
>
> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
> different daemon, which serves the historic information of MapReduce jobs
> only.
>
>
> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
>
>> Hi Experts,
>>
>> I am confusing on these two concepts. Could you help explain the
>> differences?
>>
>> Thanks!
>>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by sam liu <sa...@gmail.com>.
Zhijie,
I am much clear now. Thanks a lot!
As my understanding, besides previous Job History Server, hadoop now has a
new timeline server which could restore both the generic YARN application
history and the framework specific information. However, I think the
timeline server also include the functions of Job History Server, because
it can store the framework specific information(of course, include
mapreduce framework). In another words, Job History Server is not necessary
any more.* If that's the case, why hadoop still include Job History Server?*
2014-04-23 12:56 GMT+08:00 Zhijie Shen <zs...@hortonworks.com>:
> In Hadoop 2.4, we have delivered the timeline server at a preview stage,
> which actually can serve some generic YARN application history as well as
> the framework specific information. Due to the development logistics, we
> have created the two concepts: History Server and Timeline Server. To be
> simple, you can consider the history server of the service of the generic
> YARN application information, while consider the timeline server of the
> service of the framework specific information. Importantly, we just have
> one daemon, which includes both services, and which we'd like to call
> timeline server (unfortunately, the confusing thing is that the command to
> start the daemon is "historyserver"). We're going on working on the
> timeline server to integrate these two parts, including refactoring the
> names.
>
> BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
> different daemon, which serves the historic information of MapReduce jobs
> only.
>
>
> On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
>
>> Hi Experts,
>>
>> I am confusing on these two concepts. Could you help explain the
>> differences?
>>
>> Thanks!
>>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
In Hadoop 2.4, we have delivered the timeline server at a preview stage,
which actually can serve some generic YARN application history as well as
the framework specific information. Due to the development logistics, we
have created the two concepts: History Server and Timeline Server. To be
simple, you can consider the history server of the service of the generic
YARN application information, while consider the timeline server of the
service of the framework specific information. Importantly, we just have
one daemon, which includes both services, and which we'd like to call
timeline server (unfortunately, the confusing thing is that the command to
start the daemon is "historyserver"). We're going on working on the
timeline server to integrate these two parts, including refactoring the
names.
BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
different daemon, which serves the historic information of MapReduce jobs
only.
On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
> Hi Experts,
>
> I am confusing on these two concepts. Could you help explain the
> differences?
>
> Thanks!
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
In Hadoop 2.4, we have delivered the timeline server at a preview stage,
which actually can serve some generic YARN application history as well as
the framework specific information. Due to the development logistics, we
have created the two concepts: History Server and Timeline Server. To be
simple, you can consider the history server of the service of the generic
YARN application information, while consider the timeline server of the
service of the framework specific information. Importantly, we just have
one daemon, which includes both services, and which we'd like to call
timeline server (unfortunately, the confusing thing is that the command to
start the daemon is "historyserver"). We're going on working on the
timeline server to integrate these two parts, including refactoring the
names.
BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
different daemon, which serves the historic information of MapReduce jobs
only.
On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
> Hi Experts,
>
> I am confusing on these two concepts. Could you help explain the
> differences?
>
> Thanks!
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
In Hadoop 2.4, we have delivered the timeline server at a preview stage,
which actually can serve some generic YARN application history as well as
the framework specific information. Due to the development logistics, we
have created the two concepts: History Server and Timeline Server. To be
simple, you can consider the history server of the service of the generic
YARN application information, while consider the timeline server of the
service of the framework specific information. Importantly, we just have
one daemon, which includes both services, and which we'd like to call
timeline server (unfortunately, the confusing thing is that the command to
start the daemon is "historyserver"). We're going on working on the
timeline server to integrate these two parts, including refactoring the
names.
BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
different daemon, which serves the historic information of MapReduce jobs
only.
On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
> Hi Experts,
>
> I am confusing on these two concepts. Could you help explain the
> differences?
>
> Thanks!
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Differences between HistoryServer and Yarn TimeLine server?
Posted by Zhijie Shen <zs...@hortonworks.com>.
In Hadoop 2.4, we have delivered the timeline server at a preview stage,
which actually can serve some generic YARN application history as well as
the framework specific information. Due to the development logistics, we
have created the two concepts: History Server and Timeline Server. To be
simple, you can consider the history server of the service of the generic
YARN application information, while consider the timeline server of the
service of the framework specific information. Importantly, we just have
one daemon, which includes both services, and which we'd like to call
timeline server (unfortunately, the confusing thing is that the command to
start the daemon is "historyserver"). We're going on working on the
timeline server to integrate these two parts, including refactoring the
names.
BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a
different daemon, which serves the historic information of MapReduce jobs
only.
On Tue, Apr 22, 2014 at 8:44 PM, sam liu <sa...@gmail.com> wrote:
> Hi Experts,
>
> I am confusing on these two concepts. Could you help explain the
> differences?
>
> Thanks!
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.