You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Timothy Chan <tc...@lumoslabs.com> on 2017/03/06 04:18:00 UTC

How do I deal with ever growing application log

I'm running a single worker EMR cluster for a Structured Streaming job. How
do I deal with my application log filling up HDFS?

/var/log/spark/apps/application_1487823545416_0021_1.inprogress

is currently 21.8 GB

*Sent with Shift
<https://tryshift.com/?utm_source=SentWithShift&utm_campaign=Sent%20with%20Shift%20Signature&utm_medium=Email%20Signature&utm_content=General%20Email%20Group/>*

Re: How do I deal with ever growing application log

Posted by Noorul Islam Kamal Malmiyoda <no...@noorul.com>.
Or you could use sinks like elasticsearch.

Regards,
Noorul

On Mon, Mar 6, 2017 at 10:52 AM, devjyoti patra <dj...@gmail.com> wrote:
> Timothy, why are you writing application logs to HDFS? In case you want to
> analyze these logs later, you can write to local storage on your slave nodes
> and later rotate those files to a suitable location. If they are only going
> to useful for debugging the application, you can always remove them
> periodically.
> Thanks,
> Dev
>
> On Mar 6, 2017 9:48 AM, "Timothy Chan" <tc...@lumoslabs.com> wrote:
>>
>> I'm running a single worker EMR cluster for a Structured Streaming job.
>> How do I deal with my application log filling up HDFS?
>>
>> /var/log/spark/apps/application_1487823545416_0021_1.inprogress
>>
>> is currently 21.8 GB
>>
>> Sent with Shift

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: How do I deal with ever growing application log

Posted by devjyoti patra <dj...@gmail.com>.
Timothy, why are you writing application logs to HDFS? In case you want to
analyze these logs later, you can write to local storage on your slave
nodes and later rotate those files to a suitable location. If they are only
going to useful for debugging the application, you can always remove them
periodically.
Thanks,
Dev

On Mar 6, 2017 9:48 AM, "Timothy Chan" <tc...@lumoslabs.com> wrote:

> I'm running a single worker EMR cluster for a Structured Streaming job.
> How do I deal with my application log filling up HDFS?
>
> /var/log/spark/apps/application_1487823545416_0021_1.inprogress
>
> is currently 21.8 GB
>
> *Sent with Shift
> <https://tryshift.com/?utm_source=SentWithShift&utm_campaign=Sent%20with%20Shift%20Signature&utm_medium=Email%20Signature&utm_content=General%20Email%20Group/>*
>