You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Yuanjian Li <xy...@gmail.com> on 2023/07/14 17:31:18 UTC
[Reminder] Spark 3.5 Branch Cut
Hi everyone,
As discussed earlier in "Time for Spark v3.5.0 release", I will cut
branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.
Please plan your PR merge accordingly with the given timeline. Currently,
we have received the following exception merge requests:
- SPARK-44421: Reattach to existing execute in Spark Connect (server
mechanism)
- SPARK-44423: Reattach to existing execute in Spark Connect (scala
client)
- SPARK-44424: Reattach to existing execute in Spark Connect (python
client)
If there are any other exception feature requests, please reply to this
email. We will not merge any new features in 3.5 after the branch cut.
Best,
Yuanjian
Re: [Reminder] Spark 3.5 Branch Cut
Posted by Herman van Hovell <he...@databricks.com.INVALID>.
Hi Yuanjian,
For the ongoing encoder work for the connect scala client I'd like to get
the following tickets in:
- SPARK-44396 <https://issues.apache.org/jira/browse/SPARK-44396> :
Direct Arrow Deserialization
- SPARK-44449 <https://issues.apache.org/jira/browse/SPARK-44449> :
Upcasting for Arrow Deserialization
- SPARK-44450 <https://issues.apache.org/jira/browse/SPARK-44450> : Make
direct Arrow encoding work with SQL/API.
Cheers,
Herman
On Sat, Jul 15, 2023 at 7:53 AM Enrico Minack <in...@enrico.minack.dev>
wrote:
> Speaking of JdbcDialect, is there any interest in getting upserts for JDBC
> into 3.5.0?
>
> [SPARK-19335][SPARK-38200][SQL] Add upserts for writing to JDBC:
> https://github.com/apache/spark/pull/41518
> [SPARK-19335][SPARK-38200][SQL] Add upserts for writing to JDBC using
> MERGE INTO with temp table: https://github.com/apache/spark/pull/41611
>
> Enrico
>
>
> Am 15.07.23 um 04:10 schrieb Jia Fan:
>
> Can we put [SPARK-44262][SQL] Add `dropTable` and `getInsertStatement` to
> JdbcDialect into 3.5.0?
> https://github.com/apache/spark/pull/41855
> Since this is the last major version update of 3.x, I think we need to
> make sure JdbcDialect can support more databases.
>
>
> Gengliang Wang <lt...@gmail.com> 于2023年7月15日周六 05:20写道：
>
>> Hi Yuanjian,
>>
>> Besides the abovementioned changes, it would be great to include the UI
>> page for Spakr Connect: SPARK-44394
>> <https://issues.apache.org/jira/browse/SPARK-44394>.
>>
>> Best Regards,
>> Gengliang
>>
>> On Fri, Jul 14, 2023 at 11:44 AM Julek Sompolski
>> <ju...@databricks.com.invalid> <ju...@databricks.com.invalid> wrote:
>>
>>> Thank you,
>>> My changes that you listed are tracked under this Epic:
>>> https://issues.apache.org/jira/browse/SPARK-43754
>>> I am also working on https://issues.apache.org/jira/browse/SPARK-44422,
>>> didn't mention it before because I have hopes that this one will make it
>>> before the cut.
>>>
>>> (Unrelated) My colleague is also working on
>>> https://issues.apache.org/jira/browse/SPARK-43923 and I am reviewing
>>> https://github.com/apache/spark/pull/41443, so I hope that that one
>>> will also make it before the cut.
>>>
>>> Best regards,
>>> Juliusz Sompolski
>>>
>>> On Fri, Jul 14, 2023 at 7:34 PM Yuanjian Li <xy...@gmail.com>
>>> wrote:
>>>
>>>> Hi everyone,
>>>> As discussed earlier in "Time for Spark v3.5.0 release", I will cut
>>>> branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.
>>>>
>>>> Please plan your PR merge accordingly with the given timeline.
>>>> Currently, we have received the following exception merge requests:
>>>>
>>>> - SPARK-44421: Reattach to existing execute in Spark Connect
>>>> (server mechanism)
>>>> - SPARK-44423: Reattach to existing execute in Spark Connect
>>>> (scala client)
>>>> - SPARK-44424: Reattach to existing execute in Spark Connect
>>>> (python client)
>>>>
>>>> If there are any other exception feature requests, please reply to this
>>>> email. We will not merge any new features in 3.5 after the branch cut.
>>>>
>>>> Best,
>>>> Yuanjian
>>>>
>>>
>
Re: [Reminder] Spark 3.5 Branch Cut
Posted by Enrico Minack <in...@enrico.minack.dev>.
Speaking of JdbcDialect, is there any interest in getting upserts for
JDBC into 3.5.0?
[SPARK-19335][SPARK-38200][SQL] Add upserts for writing to JDBC:
https://github.com/apache/spark/pull/41518
[SPARK-19335][SPARK-38200][SQL] Add upserts for writing to JDBC using
MERGE INTO with temp table: https://github.com/apache/spark/pull/41611
Enrico
Am 15.07.23 um 04:10 schrieb Jia Fan:
> Can we put [SPARK-44262][SQL] Add `dropTable` and `getInsertStatement`
> to JdbcDialect into 3.5.0?
> https://github.com/apache/spark/pull/41855
> Since this is the last major version update of 3.x, I think we need to
> make sure JdbcDialect can support more databases.
>
>
> Gengliang Wang <lt...@gmail.com> 于2023年7月15日周六 05:20写道：
>
> Hi Yuanjian,
>
> Besides the abovementioned changes, it would be great to include
> the UI page for Spakr Connect: SPARK-44394
> <https://issues.apache.org/jira/browse/SPARK-44394>.
>
> Best Regards,
> Gengliang
>
> On Fri, Jul 14, 2023 at 11:44 AM Julek Sompolski
> <ju...@databricks.com.invalid> wrote:
>
> Thank you,
> My changes that you listed are tracked under this Epic:
> https://issues.apache.org/jira/browse/SPARK-43754
> I am also working on
> https://issues.apache.org/jira/browse/SPARK-44422, didn't
> mention it before because I have hopes that this one will make
> it before the cut.
>
> (Unrelated) My colleague is also working on
> https://issues.apache.org/jira/browse/SPARK-43923 and I am
> reviewing https://github.com/apache/spark/pull/41443, so I
> hope that that one will also make it before the cut.
>
> Best regards,
> Juliusz Sompolski
>
> On Fri, Jul 14, 2023 at 7:34 PM Yuanjian Li
> <xy...@gmail.com> wrote:
>
> Hi everyone,
> As discussed earlier in "Time for Spark v3.5.0 release", I
> will cut branch-3.5 on *Monday, July 17th at 1 pm PST* as
> scheduled.
>
> Please plan your PR merge accordingly with the given
> timeline. Currently, we have received the following
> exception merge requests:
>
> * SPARK-44421: Reattach to existing execute in Spark
> Connect (server mechanism)
> * SPARK-44423: Reattach to existing execute in Spark
> Connect (scala client)
> * SPARK-44424: Reattach to existing execute in Spark
> Connect (python client)
>
> If there are any other exception feature requests, please
> reply to this email. We will not merge any new features in
> 3.5 after the branch cut.
>
> Best,
> Yuanjian
>
Re: [Reminder] Spark 3.5 Branch Cut
Posted by Jia Fan <fa...@gmail.com>.
Can we put [SPARK-44262][SQL] Add `dropTable` and `getInsertStatement` to
JdbcDialect into 3.5.0?
https://github.com/apache/spark/pull/41855
Since this is the last major version update of 3.x, I think we need to make
sure JdbcDialect can support more databases.
Gengliang Wang <lt...@gmail.com> 于2023年7月15日周六 05:20写道：
> Hi Yuanjian,
>
> Besides the abovementioned changes, it would be great to include the UI
> page for Spakr Connect: SPARK-44394
> <https://issues.apache.org/jira/browse/SPARK-44394>.
>
> Best Regards,
> Gengliang
>
> On Fri, Jul 14, 2023 at 11:44 AM Julek Sompolski
> <ju...@databricks.com.invalid> wrote:
>
>> Thank you,
>> My changes that you listed are tracked under this Epic:
>> https://issues.apache.org/jira/browse/SPARK-43754
>> I am also working on https://issues.apache.org/jira/browse/SPARK-44422,
>> didn't mention it before because I have hopes that this one will make it
>> before the cut.
>>
>> (Unrelated) My colleague is also working on
>> https://issues.apache.org/jira/browse/SPARK-43923 and I am reviewing
>> https://github.com/apache/spark/pull/41443, so I hope that that one will
>> also make it before the cut.
>>
>> Best regards,
>> Juliusz Sompolski
>>
>> On Fri, Jul 14, 2023 at 7:34 PM Yuanjian Li <xy...@gmail.com>
>> wrote:
>>
>>> Hi everyone,
>>> As discussed earlier in "Time for Spark v3.5.0 release", I will cut
>>> branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.
>>>
>>> Please plan your PR merge accordingly with the given timeline.
>>> Currently, we have received the following exception merge requests:
>>>
>>> - SPARK-44421: Reattach to existing execute in Spark Connect (server
>>> mechanism)
>>> - SPARK-44423: Reattach to existing execute in Spark Connect (scala
>>> client)
>>> - SPARK-44424: Reattach to existing execute in Spark Connect
>>> (python client)
>>>
>>> If there are any other exception feature requests, please reply to this
>>> email. We will not merge any new features in 3.5 after the branch cut.
>>>
>>> Best,
>>> Yuanjian
>>>
>>
Re: [Reminder] Spark 3.5 Branch Cut
Posted by Gengliang Wang <lt...@gmail.com>.
Hi Yuanjian,
Besides the abovementioned changes, it would be great to include the UI
page for Spakr Connect: SPARK-44394
<https://issues.apache.org/jira/browse/SPARK-44394>.
Best Regards,
Gengliang
On Fri, Jul 14, 2023 at 11:44 AM Julek Sompolski
<ju...@databricks.com.invalid> wrote:
> Thank you,
> My changes that you listed are tracked under this Epic:
> https://issues.apache.org/jira/browse/SPARK-43754
> I am also working on https://issues.apache.org/jira/browse/SPARK-44422,
> didn't mention it before because I have hopes that this one will make it
> before the cut.
>
> (Unrelated) My colleague is also working on
> https://issues.apache.org/jira/browse/SPARK-43923 and I am reviewing
> https://github.com/apache/spark/pull/41443, so I hope that that one will
> also make it before the cut.
>
> Best regards,
> Juliusz Sompolski
>
> On Fri, Jul 14, 2023 at 7:34 PM Yuanjian Li <xy...@gmail.com>
> wrote:
>
>> Hi everyone,
>> As discussed earlier in "Time for Spark v3.5.0 release", I will cut
>> branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.
>>
>> Please plan your PR merge accordingly with the given timeline. Currently,
>> we have received the following exception merge requests:
>>
>> - SPARK-44421: Reattach to existing execute in Spark Connect (server
>> mechanism)
>> - SPARK-44423: Reattach to existing execute in Spark Connect (scala
>> client)
>> - SPARK-44424: Reattach to existing execute in Spark Connect (python
>> client)
>>
>> If there are any other exception feature requests, please reply to this
>> email. We will not merge any new features in 3.5 after the branch cut.
>>
>> Best,
>> Yuanjian
>>
>
Re: [Reminder] Spark 3.5 Branch Cut
Posted by Julek Sompolski <ju...@databricks.com.INVALID>.
Thank you,
My changes that you listed are tracked under this Epic:
https://issues.apache.org/jira/browse/SPARK-43754
I am also working on https://issues.apache.org/jira/browse/SPARK-44422,
didn't mention it before because I have hopes that this one will make it
before the cut.
(Unrelated) My colleague is also working on
https://issues.apache.org/jira/browse/SPARK-43923 and I am reviewing
https://github.com/apache/spark/pull/41443, so I hope that that one will
also make it before the cut.
Best regards,
Juliusz Sompolski
On Fri, Jul 14, 2023 at 7:34 PM Yuanjian Li <xy...@gmail.com> wrote:
> Hi everyone,
> As discussed earlier in "Time for Spark v3.5.0 release", I will cut
> branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.
>
> Please plan your PR merge accordingly with the given timeline. Currently,
> we have received the following exception merge requests:
>
> - SPARK-44421: Reattach to existing execute in Spark Connect (server
> mechanism)
> - SPARK-44423: Reattach to existing execute in Spark Connect (scala
> client)
> - SPARK-44424: Reattach to existing execute in Spark Connect (python
> client)
>
> If there are any other exception feature requests, please reply to this
> email. We will not merge any new features in 3.5 after the branch cut.
>
> Best,
> Yuanjian
>
Re: [Reminder] Spark 3.5 Branch Cut
Posted by Raghu Angadi <ra...@databricks.com.INVALID>.
Thank you. We plan to get remaining major pieces for Streaming Spark
Connect (Epic SPARK-42938
<https://issues.apache.org/jira/browse/SPARK-42938>).
I would like to request a warmfix exception for the following tweaks and
improvements over the next two weeks (all in the same epic).
- SPARK-42944 : (Will mostly hit Monday deadline, just in case) Python
foreachBatch
- SPARK-42941 : (WIP, but might slip Monday deadline): Python streaming
listener
- SPARK-44400 : Improve session access in connect Scala
StreamingQueryListener
- SPARK-44432 : Allow timeout of sessions when client disconnects and
terminate queries
- SPARK-44433 : Improve termination logic for Python processes for
foreachBatch & query listener
- SPARK-44434 : More Scala tests for foreachBatch & query listener
- SPARK-44435 : More Python tests for foreachBatch & query listener
- SPARK-44436 : Use Connect DataFrame for Scala foreachBatch in Connect
On Fri, Jul 14, 2023 at 10:33 AM Yuanjian Li <xy...@gmail.com> wrote:
> Hi everyone,
> As discussed earlier in "Time for Spark v3.5.0 release", I will cut
> branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled.
>
> Please plan your PR merge accordingly with the given timeline. Currently,
> we have received the following exception merge requests:
>
> - SPARK-44421: Reattach to existing execute in Spark Connect (server
> mechanism)
> - SPARK-44423: Reattach to existing execute in Spark Connect (scala
> client)
> - SPARK-44424: Reattach to existing execute in Spark Connect (python
> client)
>
> If there are any other exception feature requests, please reply to this
> email. We will not merge any new features in 3.5 after the branch cut.
>
> Best,
> Yuanjian
>