You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by allonsy <lu...@gmail.com> on 2015/08/13 16:50:29 UTC
spark.streaming.maxRatePerPartition parameter: what are the
benefits?
Hello everyone,
in the new Kafka Direct API, what are the benefits of setting a value for
*spark.streaming.maxRatePerPartition*?
In my case, I have 2 seconds batches consuming ~15k tuples from a topic
split into 48 partitions (4 workers, 16 total cores).
Is there any particular value I should be setting the parameter to, in order
to achieve better performances? And what happens if I don't set the value at
all?
I could not find any detailed explanation about this.
Thank you!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-maxRatePerPartition-parameter-what-are-the-benefits-tp24241.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark.streaming.maxRatePerPartition parameter: what are the benefits?
Posted by Cody Koeninger <co...@koeninger.org>.
It's just to limit the maximum number of records a given executor needs to
deal with in a given batch.
Typical usage would be if you're starting a stream from the beginning of a
kafka log, or after a long downtime, and don't want ALL of the messages in
the first batch.
On Thu, Aug 13, 2015 at 8:50 AM, allonsy <lu...@gmail.com> wrote:
> Hello everyone,
>
> in the new Kafka Direct API, what are the benefits of setting a value for
> *spark.streaming.maxRatePerPartition*?
>
> In my case, I have 2 seconds batches consuming ~15k tuples from a topic
> split into 48 partitions (4 workers, 16 total cores).
>
> Is there any particular value I should be setting the parameter to, in
> order
> to achieve better performances? And what happens if I don't set the value
> at
> all?
>
> I could not find any detailed explanation about this.
>
> Thank you!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-maxRatePerPartition-parameter-what-are-the-benefits-tp24241.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>