You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Timothy Chan <tc...@lumoslabs.com> on 2017/03/06 04:18:00 UTC
How do I deal with ever growing application log
I'm running a single worker EMR cluster for a Structured Streaming job. How
do I deal with my application log filling up HDFS?
/var/log/spark/apps/application_1487823545416_0021_1.inprogress
is currently 21.8 GB
*Sent with Shift
<https://tryshift.com/?utm_source=SentWithShift&utm_campaign=Sent%20with%20Shift%20Signature&utm_medium=Email%20Signature&utm_content=General%20Email%20Group/>*
Re: How do I deal with ever growing application log
Posted by Noorul Islam Kamal Malmiyoda <no...@noorul.com>.
Or you could use sinks like elasticsearch.
Regards,
Noorul
On Mon, Mar 6, 2017 at 10:52 AM, devjyoti patra <dj...@gmail.com> wrote:
> Timothy, why are you writing application logs to HDFS? In case you want to
> analyze these logs later, you can write to local storage on your slave nodes
> and later rotate those files to a suitable location. If they are only going
> to useful for debugging the application, you can always remove them
> periodically.
> Thanks,
> Dev
>
> On Mar 6, 2017 9:48 AM, "Timothy Chan" <tc...@lumoslabs.com> wrote:
>>
>> I'm running a single worker EMR cluster for a Structured Streaming job.
>> How do I deal with my application log filling up HDFS?
>>
>> /var/log/spark/apps/application_1487823545416_0021_1.inprogress
>>
>> is currently 21.8 GB
>>
>> Sent with Shift
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: How do I deal with ever growing application log
Posted by devjyoti patra <dj...@gmail.com>.
Timothy, why are you writing application logs to HDFS? In case you want to
analyze these logs later, you can write to local storage on your slave
nodes and later rotate those files to a suitable location. If they are only
going to useful for debugging the application, you can always remove them
periodically.
Thanks,
Dev
On Mar 6, 2017 9:48 AM, "Timothy Chan" <tc...@lumoslabs.com> wrote:
> I'm running a single worker EMR cluster for a Structured Streaming job.
> How do I deal with my application log filling up HDFS?
>
> /var/log/spark/apps/application_1487823545416_0021_1.inprogress
>
> is currently 21.8 GB
>
> *Sent with Shift
> <https://tryshift.com/?utm_source=SentWithShift&utm_campaign=Sent%20with%20Shift%20Signature&utm_medium=Email%20Signature&utm_content=General%20Email%20Group/>*
>