Re: Re: spark+kafka+dynamic resource allocation

[Mich Talebzadeh <mi...@gmail.com>]

Sure, I suggest that you add a note to that Jira and express your interest.

HTH



   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>


 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Mon, 30 Jan 2023 at 01:56, Lingzhe Sun <li...@hirain.com> wrote:

> Hi Mich，
>
> Thanks for the information. I myself think this open issue should have
> higher priority as more streaming application is being built using spark.
> Hope this becomes a feature in the coming future.
>
> ------------------------------
> BR
> Lingzhe Sun
>
>
> *From:* Mich Talebzadeh <mi...@gmail.com>
> *Date:* 2023-01-30 02:14
> *To:* Lingzhe Sun <li...@hirain.com>
> *CC:* ashok34668@yahoo.com; User <us...@spark.apache.org>
> *Subject:* Re: Re: spark+kafka+dynamic resource allocation
> Hi,
>
> Spark Structured Streaming currently does not support dynamic allocation
> (see SPARK-24815: Structured Streaming should support dynamic allocation
> <https://issues.apache.org/jira/browse/SPARK-24815>). which is still open
>
>
> Autoscaling in Cloud offerings
> <https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/autoscaling#autoscaling_and_spark_streaming>like
> Google Dataproc does not support Spark Structured Streaming either
>
>
> HTH
>
>
>    view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sun, 29 Jan 2023 at 01:49, Lingzhe Sun <li...@hirain.com> wrote:
>
>> Thank you for the response. But the reference does not seem to be
>> answering any of those questions.
>>
>> BS
>> Lingzhe Sun
>>
>>
>> *From:* ashok34668@yahoo.com
>> *Date:* 2023-01-29 04:01
>> *To:* User <us...@spark.apache.org>; Lingzhe Sun <li...@hirain.com>
>> *Subject:* Re: spark+kafka+dynamic resource allocation
>> Hi,
>>
>> Worth checking this link
>>
>>
>> https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation
>>
>> On Saturday, 28 January 2023 at 06:18:28 GMT, Lingzhe Sun <
>> lingzhe.sun@hirain.com> wrote:
>>
>>
>> Hi all,
>>
>> I'm wondering if dynamic resource allocation works in spark+kafka
>> streaming applications. Here're some questions:
>>
>>    - Will structured streaming be supported?
>>    - Is the number of consumers always equal to the number of the
>>    partitions of subscribed topic (let's say there's only one topic)?
>>    - If consumers is evenly distributed across executors, will newly
>>    added executor(through dynamic resource allocation) trigger a consumer
>>    reassignment?
>>    - Would it be simply a bad idea to use dynamic resource allocation in
>>    streaming app, because there's no way to scale down number of executors
>>    unless no data is coming in?
>>
>> Any thoughts are welcomed.
>>
>> Lingzhe Sun
>> Hirain Technology
>>
>>