log4j Spark-worker performance problem

[vaibhavrtk <va...@gmail.com>]

Hello

We need a lot of logging for our application about 1000 lines needed to be
logged per message we process and we process 1000 msgs/sec. So total lines
needed to be logged is /1000*1000/sec/. As it is going to be written in a
file. Will writing so much logs will impact the processing power of spark by
a lot?
If yes, What can be the alternative?

Note: This much logging is required for the appropriate monitoring of the
application.
Let me know if more information is needed.

Thanks
Vaibhav




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/log4j-Spark-worker-performance-problem-tp24842.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: log4j Spark-worker performance problem

[Akhil Das <ak...@sigmoidanalytics.com>]

Depends how big the lines are, on a typical HDD you can write at max
10-15MB/s, and on SSDs it can be upto 30-40MB/s.

Thanks
Best Regards

On Mon, Sep 28, 2015 at 3:57 PM, vaibhavrtk <va...@gmail.com> wrote:

> Hello
>
> We need a lot of logging for our application about 1000 lines needed to be
> logged per message we process and we process 1000 msgs/sec. So total lines
> needed to be logged is /1000*1000/sec/. As it is going to be written in a
> file. Will writing so much logs will impact the processing power of spark
> by
> a lot?
> If yes, What can be the alternative?
>
> Note: This much logging is required for the appropriate monitoring of the
> application.
> Let me know if more information is needed.
>
> Thanks
> Vaibhav
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/log4j-Spark-worker-performance-problem-tp24842.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>