You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Emmanuel Espina <es...@gmail.com> on 2014/02/10 20:49:22 UTC
Can Yarn AppMaster move Container logs to hdfs?
I'm building a custom YARN app where the App Master is a log running
service that can start jobs in containers.
The ideal situation for us would be to be able to move all the logs
produced by each container to hdfs. The aggregation options that yarn
provides does this but only after the app master finishes (and in our case
it never finishes).
Is there any way of doing this?
Thanks
Emmanuel
Re: Can Yarn AppMaster move Container logs to hdfs?
Posted by Jian He <jh...@hortonworks.com>.
Hi Emmanuel, log aggregation now only aggregates finished apps logs onto
hdfs. there's no way as of now to support running apps. that'll be a to-do
feature in the feature.
Jian
On Mon, Feb 10, 2014 at 11:53 AM, Emmanuel Espina
<es...@gmail.com>wrote:
> Sorry when I said log running I meant LONG running, that is, a service,
> not a batch job
>
>
> 2014-02-10 16:49 GMT-03:00 Emmanuel Espina <es...@gmail.com>:
>
> I'm building a custom YARN app where the App Master is a log running
>> service that can start jobs in containers.
>>
>> The ideal situation for us would be to be able to move all the logs
>> produced by each container to hdfs. The aggregation options that yarn
>> provides does this but only after the app master finishes (and in our case
>> it never finishes).
>>
>> Is there any way of doing this?
>>
>> Thanks
>> Emmanuel
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Can Yarn AppMaster move Container logs to hdfs?
Posted by Jian He <jh...@hortonworks.com>.
Hi Emmanuel, log aggregation now only aggregates finished apps logs onto
hdfs. there's no way as of now to support running apps. that'll be a to-do
feature in the feature.
Jian
On Mon, Feb 10, 2014 at 11:53 AM, Emmanuel Espina
<es...@gmail.com>wrote:
> Sorry when I said log running I meant LONG running, that is, a service,
> not a batch job
>
>
> 2014-02-10 16:49 GMT-03:00 Emmanuel Espina <es...@gmail.com>:
>
> I'm building a custom YARN app where the App Master is a log running
>> service that can start jobs in containers.
>>
>> The ideal situation for us would be to be able to move all the logs
>> produced by each container to hdfs. The aggregation options that yarn
>> provides does this but only after the app master finishes (and in our case
>> it never finishes).
>>
>> Is there any way of doing this?
>>
>> Thanks
>> Emmanuel
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Can Yarn AppMaster move Container logs to hdfs?
Posted by Jian He <jh...@hortonworks.com>.
Hi Emmanuel, log aggregation now only aggregates finished apps logs onto
hdfs. there's no way as of now to support running apps. that'll be a to-do
feature in the feature.
Jian
On Mon, Feb 10, 2014 at 11:53 AM, Emmanuel Espina
<es...@gmail.com>wrote:
> Sorry when I said log running I meant LONG running, that is, a service,
> not a batch job
>
>
> 2014-02-10 16:49 GMT-03:00 Emmanuel Espina <es...@gmail.com>:
>
> I'm building a custom YARN app where the App Master is a log running
>> service that can start jobs in containers.
>>
>> The ideal situation for us would be to be able to move all the logs
>> produced by each container to hdfs. The aggregation options that yarn
>> provides does this but only after the app master finishes (and in our case
>> it never finishes).
>>
>> Is there any way of doing this?
>>
>> Thanks
>> Emmanuel
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Can Yarn AppMaster move Container logs to hdfs?
Posted by Jian He <jh...@hortonworks.com>.
Hi Emmanuel, log aggregation now only aggregates finished apps logs onto
hdfs. there's no way as of now to support running apps. that'll be a to-do
feature in the feature.
Jian
On Mon, Feb 10, 2014 at 11:53 AM, Emmanuel Espina
<es...@gmail.com>wrote:
> Sorry when I said log running I meant LONG running, that is, a service,
> not a batch job
>
>
> 2014-02-10 16:49 GMT-03:00 Emmanuel Espina <es...@gmail.com>:
>
> I'm building a custom YARN app where the App Master is a log running
>> service that can start jobs in containers.
>>
>> The ideal situation for us would be to be able to move all the logs
>> produced by each container to hdfs. The aggregation options that yarn
>> provides does this but only after the app master finishes (and in our case
>> it never finishes).
>>
>> Is there any way of doing this?
>>
>> Thanks
>> Emmanuel
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Can Yarn AppMaster move Container logs to hdfs?
Posted by Emmanuel Espina <es...@gmail.com>.
Sorry when I said log running I meant LONG running, that is, a service, not
a batch job
2014-02-10 16:49 GMT-03:00 Emmanuel Espina <es...@gmail.com>:
> I'm building a custom YARN app where the App Master is a log running
> service that can start jobs in containers.
>
> The ideal situation for us would be to be able to move all the logs
> produced by each container to hdfs. The aggregation options that yarn
> provides does this but only after the app master finishes (and in our case
> it never finishes).
>
> Is there any way of doing this?
>
> Thanks
> Emmanuel
>
Re: Can Yarn AppMaster move Container logs to hdfs?
Posted by Emmanuel Espina <es...@gmail.com>.
Sorry when I said log running I meant LONG running, that is, a service, not
a batch job
2014-02-10 16:49 GMT-03:00 Emmanuel Espina <es...@gmail.com>:
> I'm building a custom YARN app where the App Master is a log running
> service that can start jobs in containers.
>
> The ideal situation for us would be to be able to move all the logs
> produced by each container to hdfs. The aggregation options that yarn
> provides does this but only after the app master finishes (and in our case
> it never finishes).
>
> Is there any way of doing this?
>
> Thanks
> Emmanuel
>
Re: Can Yarn AppMaster move Container logs to hdfs?
Posted by Emmanuel Espina <es...@gmail.com>.
Sorry when I said log running I meant LONG running, that is, a service, not
a batch job
2014-02-10 16:49 GMT-03:00 Emmanuel Espina <es...@gmail.com>:
> I'm building a custom YARN app where the App Master is a log running
> service that can start jobs in containers.
>
> The ideal situation for us would be to be able to move all the logs
> produced by each container to hdfs. The aggregation options that yarn
> provides does this but only after the app master finishes (and in our case
> it never finishes).
>
> Is there any way of doing this?
>
> Thanks
> Emmanuel
>
Re: Can Yarn AppMaster move Container logs to hdfs?
Posted by Emmanuel Espina <es...@gmail.com>.
Sorry when I said log running I meant LONG running, that is, a service, not
a batch job
2014-02-10 16:49 GMT-03:00 Emmanuel Espina <es...@gmail.com>:
> I'm building a custom YARN app where the App Master is a log running
> service that can start jobs in containers.
>
> The ideal situation for us would be to be able to move all the logs
> produced by each container to hdfs. The aggregation options that yarn
> provides does this but only after the app master finishes (and in our case
> it never finishes).
>
> Is there any way of doing this?
>
> Thanks
> Emmanuel
>