You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Gengliang Wang <lt...@gmail.com> on 2021/08/10 08:11:58 UTC

Spark 3.2.0 first RC next week

Hi all,

As of now, there are still some open/in-progress blockers for Spark 3.2.0
release:

   - Prohibit update mode in native support of session window (SPARK-36463
   <https://issues.apache.org/jira/browse/SPARK-36463>)
   - Avoid inlining non-deterministic With-CTEs(SPARK-36447
   <https://issues.apache.org/jira/browse/SPARK-36447>)
   - Data Source V2: Remove read specific distributions(SPARK-33807
   <https://issues.apache.org/jira/browse/SPARK-33807>)
   - Support fetching shuffle blocks in batch with i/o encryption(
   SPARK-34827 <https://issues.apache.org/jira/browse/SPARK-34827>)
   - Add a new Maven profile "no-shaded-client" for older Hadoop 3.x
   versions(SPARK-35959 <https://issues.apache.org/jira/browse/SPARK-35959>)
   - Review and fix issues in API docs(SPARK-34185
   <https://issues.apache.org/jira/browse/SPARK-34185>)
   - Introduce the RocksDBStateStoreProvider in the programming guide(
   SPARK-36041 <https://issues.apache.org/jira/browse/SPARK-36041>)
   - Push-based shuffle documentation(SPARK-36374
   <https://issues.apache.org/jira/browse/SPARK-36374>)

Thus, I propose to cut RC1 next week after all the blockers are resolved.
If there are any other blockers, please reply to this email.

Thanks
Gengliang

Re: Spark 3.2.0 first RC next week

Posted by Gengliang Wang <lt...@gmail.com>.

Thanks for your information, Sean and Min!

On Tue, Aug 10, 2021 at 9:59 PM Min Shen <ms...@apache.org> wrote:

> Hi Gengliang,
>
> SPARK-36378 (Switch to using RPCResponse to communicate common block push
> failures to the client) should be another one.
> This introduces a slight protocol change to push-based shuffle to improve
> code robustness and performance, and is almost ready to be committed.
> Because of the protocol change, it’s best to include it with 3.2.0
> release.
>
> Best,
> Min
>
> On Tue, Aug 10, 2021 at 01:13 Gengliang Wang <lt...@gmail.com> wrote:
>
>> Hi all,
>>
>> As of now, there are still some open/in-progress blockers for Spark 3.2.0
>> release:
>>
>>    - Prohibit update mode in native support of session window (
>>    SPARK-36463 <https://issues.apache.org/jira/browse/SPARK-36463>)
>>    - Avoid inlining non-deterministic With-CTEs(SPARK-36447
>>    <https://issues.apache.org/jira/browse/SPARK-36447>)
>>    - Data Source V2: Remove read specific distributions(SPARK-33807
>>    <https://issues.apache.org/jira/browse/SPARK-33807>)
>>    - Support fetching shuffle blocks in batch with i/o encryption(
>>    SPARK-34827 <https://issues.apache.org/jira/browse/SPARK-34827>)
>>    - Add a new Maven profile "no-shaded-client" for older Hadoop 3.x
>>    versions(SPARK-35959
>>    <https://issues.apache.org/jira/browse/SPARK-35959>)
>>    - Review and fix issues in API docs(SPARK-34185
>>    <https://issues.apache.org/jira/browse/SPARK-34185>)
>>    - Introduce the RocksDBStateStoreProvider in the programming guide(
>>    SPARK-36041 <https://issues.apache.org/jira/browse/SPARK-36041>)
>>    - Push-based shuffle documentation(SPARK-36374
>>    <https://issues.apache.org/jira/browse/SPARK-36374>)
>>
>> Thus, I propose to cut RC1 next week after all the blockers are resolved.
>> If there are any other blockers, please reply to this email.
>>
>> Thanks
>> Gengliang
>>
>

Re: Spark 3.2.0 first RC next week

Posted by Min Shen <ms...@apache.org>.

Hi Gengliang,

SPARK-36378 (Switch to using RPCResponse to communicate common block push
failures to the client) should be another one.
This introduces a slight protocol change to push-based shuffle to improve
code robustness and performance, and is almost ready to be committed.
Because of the protocol change, it’s best to include it with 3.2.0 release.

Best,
Min

On Tue, Aug 10, 2021 at 01:13 Gengliang Wang <lt...@gmail.com> wrote:

> Hi all,
>
> As of now, there are still some open/in-progress blockers for Spark 3.2.0
> release:
>
>    - Prohibit update mode in native support of session window (SPARK-36463
>    <https://issues.apache.org/jira/browse/SPARK-36463>)
>    - Avoid inlining non-deterministic With-CTEs(SPARK-36447
>    <https://issues.apache.org/jira/browse/SPARK-36447>)
>    - Data Source V2: Remove read specific distributions(SPARK-33807
>    <https://issues.apache.org/jira/browse/SPARK-33807>)
>    - Support fetching shuffle blocks in batch with i/o encryption(
>    SPARK-34827 <https://issues.apache.org/jira/browse/SPARK-34827>)
>    - Add a new Maven profile "no-shaded-client" for older Hadoop 3.x
>    versions(SPARK-35959
>    <https://issues.apache.org/jira/browse/SPARK-35959>)
>    - Review and fix issues in API docs(SPARK-34185
>    <https://issues.apache.org/jira/browse/SPARK-34185>)
>    - Introduce the RocksDBStateStoreProvider in the programming guide(
>    SPARK-36041 <https://issues.apache.org/jira/browse/SPARK-36041>)
>    - Push-based shuffle documentation(SPARK-36374
>    <https://issues.apache.org/jira/browse/SPARK-36374>)
>
> Thus, I propose to cut RC1 next week after all the blockers are resolved.
> If there are any other blockers, please reply to this email.
>
> Thanks
> Gengliang
>