You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Dongjoon Hyun <do...@gmail.com> on 2025/01/24 20:24:57 UTC
FYI: SPARK-49700 Unified Scala Interface for Connect and Classic
Hi, All.
SPARK-49700 landed one hour ago.
Since this is another huge package redesign across 399 files in Spark 4.0,
please check if you are not affected accidentally.
Best Regards,
Dongjoon.
Re: FYI: SPARK-49700 Unified Scala Interface for Connect and Classic
Posted by Asif Shahid <as...@gmail.com>.
Thanks for the explanation.
Regards
Asif
On Tue, Jan 28, 2025 at 10:00 AM Herman van Hovell <he...@databricks.com>
wrote:
> There are many factors:
>
> - Typically it is a race between multiple PRs, where they all pass CI
> without the other changes, and get merged at the same time.
> - Differences between (the nightly job and the PR job) environments
> (e.g. size of the machine) can also cause these issues.
> - In the case of my PR there were some classpath subtleties
> (apparently the testing classpath is not fully hermetic), which PR testing
> did not uncover.
> - ...
>
>
> On Tue, Jan 28, 2025 at 1:53 PM Asif Shahid <as...@gmail.com> wrote:
>
>> I am genuinely curious to know, as to how do those commits which are
>> reliably failing the build, end up in master ? Is there some window of race
>> where two conflicting PRs in terms of logic ,tend to mess up the final
>> state in master ?
>> I have seen in past few months, while synching up my open PRs, fail due
>> to issues in master.
>> I suppose many times, env. issues etc can cause. But scala unit tests
>> failing on master should not be common.
>>
>> On Mon, Jan 27, 2025 at 12:37 PM Dongjoon Hyun <do...@gmail.com>
>> wrote:
>>
>>> Did you see the PR, Martin? SBT is also broken like the following and
>>> we've been waiting for actions over two days on the original PR.
>>>
>>> $ build/sbt clean "catalyst/testOnly
>>> org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite"
>>> ...
>>> [info] *** 1 SUITE ABORTED ***
>>> [error] Error during tests:
>>> [error] org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite
>>> [error] (catalyst / Test / testOnly) sbt.TestsFailedException: Tests
>>> unsuccessful
>>> [error] Total time: 66 s (01:06), completed Jan 27, 2025, 9:52:35 AM
>>>
>>> For the record, we keep it in the `master` branch still.
>>>
>>> Dongjoon.
>>>
>>>
>>>
>>>
>>> On Mon, Jan 27, 2025 at 12:33 PM Martin Grund
>>> <ma...@databricks.com.invalid> wrote:
>>>
>>>> Would it not have been mindful to wait for the original author to
>>>> investigate the PR and do a forward fix instead of reverting such a big
>>>> change? Since this was only blocking the Maven test we could have waited
>>>> probably a few more days without any issues.
>>>>
>>>> On Mon, Jan 27, 2025 at 8:32 PM Dongjoon Hyun <do...@apache.org>
>>>> wrote:
>>>>
>>>>> This is reverted from branch-4.0 via the following.
>>>>>
>>>>> - https://github.com/apache/spark/pull/49696
>>>>> Revert "[SPARK-49700][CONNECT][SQL] Unified Scala Interface for
>>>>> Connect and Classic"
>>>>>
>>>>> Dongjoon.
>>>>>
>>>>> On 2025/01/26 16:58:45 Dongjoon Hyun wrote:
>>>>> > Thank you!
>>>>> >
>>>>> > Dongjoon
>>>>> >
>>>>> > On Sat, Jan 25, 2025 at 20:01 Yang Jie <ya...@apache.org> wrote:
>>>>> >
>>>>> > > I reported a test issue that is suspected to be related to this pr：
>>>>> > >
>>>>> > > - https://github.com/apache/spark/pull/48818/files#r1929652392
>>>>> > >
>>>>> > > and it seems to be causing the failure of the Maven daily test.
>>>>> > >
>>>>> > > Thanks,
>>>>> > > Jie Yang
>>>>> > >
>>>>> > > On 2025/01/24 20:24:57 Dongjoon Hyun wrote:
>>>>> > > > Hi, All.
>>>>> > > >
>>>>> > > > SPARK-49700 landed one hour ago.
>>>>> > > >
>>>>> > > > Since this is another huge package redesign across 399 files in
>>>>> Spark
>>>>> > > 4.0,
>>>>> > > > please check if you are not affected accidentally.
>>>>> > > >
>>>>> > > > Best Regards,
>>>>> > > > Dongjoon.
>>>>> > > >
>>>>> > >
>>>>> > >
>>>>> ---------------------------------------------------------------------
>>>>> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>> > >
>>>>> > >
>>>>> >
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>
Re: FYI: SPARK-49700 Unified Scala Interface for Connect and Classic
Posted by Herman van Hovell <he...@databricks.com.INVALID>.
There are many factors:
- Typically it is a race between multiple PRs, where they all pass CI
without the other changes, and get merged at the same time.
- Differences between (the nightly job and the PR job) environments
(e.g. size of the machine) can also cause these issues.
- In the case of my PR there were some classpath subtleties (apparently
the testing classpath is not fully hermetic), which PR testing did not
uncover.
- ...
On Tue, Jan 28, 2025 at 1:53 PM Asif Shahid <as...@gmail.com> wrote:
> I am genuinely curious to know, as to how do those commits which are
> reliably failing the build, end up in master ? Is there some window of race
> where two conflicting PRs in terms of logic ,tend to mess up the final
> state in master ?
> I have seen in past few months, while synching up my open PRs, fail due to
> issues in master.
> I suppose many times, env. issues etc can cause. But scala unit tests
> failing on master should not be common.
>
> On Mon, Jan 27, 2025 at 12:37 PM Dongjoon Hyun <do...@gmail.com>
> wrote:
>
>> Did you see the PR, Martin? SBT is also broken like the following and
>> we've been waiting for actions over two days on the original PR.
>>
>> $ build/sbt clean "catalyst/testOnly
>> org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite"
>> ...
>> [info] *** 1 SUITE ABORTED ***
>> [error] Error during tests:
>> [error] org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite
>> [error] (catalyst / Test / testOnly) sbt.TestsFailedException: Tests
>> unsuccessful
>> [error] Total time: 66 s (01:06), completed Jan 27, 2025, 9:52:35 AM
>>
>> For the record, we keep it in the `master` branch still.
>>
>> Dongjoon.
>>
>>
>>
>>
>> On Mon, Jan 27, 2025 at 12:33 PM Martin Grund
>> <ma...@databricks.com.invalid> wrote:
>>
>>> Would it not have been mindful to wait for the original author to
>>> investigate the PR and do a forward fix instead of reverting such a big
>>> change? Since this was only blocking the Maven test we could have waited
>>> probably a few more days without any issues.
>>>
>>> On Mon, Jan 27, 2025 at 8:32 PM Dongjoon Hyun <do...@apache.org>
>>> wrote:
>>>
>>>> This is reverted from branch-4.0 via the following.
>>>>
>>>> - https://github.com/apache/spark/pull/49696
>>>> Revert "[SPARK-49700][CONNECT][SQL] Unified Scala Interface for Connect
>>>> and Classic"
>>>>
>>>> Dongjoon.
>>>>
>>>> On 2025/01/26 16:58:45 Dongjoon Hyun wrote:
>>>> > Thank you!
>>>> >
>>>> > Dongjoon
>>>> >
>>>> > On Sat, Jan 25, 2025 at 20:01 Yang Jie <ya...@apache.org> wrote:
>>>> >
>>>> > > I reported a test issue that is suspected to be related to this pr：
>>>> > >
>>>> > > - https://github.com/apache/spark/pull/48818/files#r1929652392
>>>> > >
>>>> > > and it seems to be causing the failure of the Maven daily test.
>>>> > >
>>>> > > Thanks,
>>>> > > Jie Yang
>>>> > >
>>>> > > On 2025/01/24 20:24:57 Dongjoon Hyun wrote:
>>>> > > > Hi, All.
>>>> > > >
>>>> > > > SPARK-49700 landed one hour ago.
>>>> > > >
>>>> > > > Since this is another huge package redesign across 399 files in
>>>> Spark
>>>> > > 4.0,
>>>> > > > please check if you are not affected accidentally.
>>>> > > >
>>>> > > > Best Regards,
>>>> > > > Dongjoon.
>>>> > > >
>>>> > >
>>>> > >
>>>> ---------------------------------------------------------------------
>>>> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> > >
>>>> > >
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
Re: FYI: SPARK-49700 Unified Scala Interface for Connect and Classic
Posted by Asif Shahid <as...@gmail.com>.
I am genuinely curious to know, as to how do those commits which are
reliably failing the build, end up in master ? Is there some window of race
where two conflicting PRs in terms of logic ,tend to mess up the final
state in master ?
I have seen in past few months, while synching up my open PRs, fail due to
issues in master.
I suppose many times, env. issues etc can cause. But scala unit tests
failing on master should not be common.
On Mon, Jan 27, 2025 at 12:37 PM Dongjoon Hyun <do...@gmail.com>
wrote:
> Did you see the PR, Martin? SBT is also broken like the following and
> we've been waiting for actions over two days on the original PR.
>
> $ build/sbt clean "catalyst/testOnly
> org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite"
> ...
> [info] *** 1 SUITE ABORTED ***
> [error] Error during tests:
> [error] org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite
> [error] (catalyst / Test / testOnly) sbt.TestsFailedException: Tests
> unsuccessful
> [error] Total time: 66 s (01:06), completed Jan 27, 2025, 9:52:35 AM
>
> For the record, we keep it in the `master` branch still.
>
> Dongjoon.
>
>
>
>
> On Mon, Jan 27, 2025 at 12:33 PM Martin Grund
> <ma...@databricks.com.invalid> wrote:
>
>> Would it not have been mindful to wait for the original author to
>> investigate the PR and do a forward fix instead of reverting such a big
>> change? Since this was only blocking the Maven test we could have waited
>> probably a few more days without any issues.
>>
>> On Mon, Jan 27, 2025 at 8:32 PM Dongjoon Hyun <do...@apache.org>
>> wrote:
>>
>>> This is reverted from branch-4.0 via the following.
>>>
>>> - https://github.com/apache/spark/pull/49696
>>> Revert "[SPARK-49700][CONNECT][SQL] Unified Scala Interface for Connect
>>> and Classic"
>>>
>>> Dongjoon.
>>>
>>> On 2025/01/26 16:58:45 Dongjoon Hyun wrote:
>>> > Thank you!
>>> >
>>> > Dongjoon
>>> >
>>> > On Sat, Jan 25, 2025 at 20:01 Yang Jie <ya...@apache.org> wrote:
>>> >
>>> > > I reported a test issue that is suspected to be related to this pr：
>>> > >
>>> > > - https://github.com/apache/spark/pull/48818/files#r1929652392
>>> > >
>>> > > and it seems to be causing the failure of the Maven daily test.
>>> > >
>>> > > Thanks,
>>> > > Jie Yang
>>> > >
>>> > > On 2025/01/24 20:24:57 Dongjoon Hyun wrote:
>>> > > > Hi, All.
>>> > > >
>>> > > > SPARK-49700 landed one hour ago.
>>> > > >
>>> > > > Since this is another huge package redesign across 399 files in
>>> Spark
>>> > > 4.0,
>>> > > > please check if you are not affected accidentally.
>>> > > >
>>> > > > Best Regards,
>>> > > > Dongjoon.
>>> > > >
>>> > >
>>> > > ---------------------------------------------------------------------
>>> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> > >
>>> > >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
Re: FYI: SPARK-49700 Unified Scala Interface for Connect and Classic
Posted by Dongjoon Hyun <do...@gmail.com>.
Did you see the PR, Martin? SBT is also broken like the following and we've
been waiting for actions over two days on the original PR.
$ build/sbt clean "catalyst/testOnly
org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite"
...
[info] *** 1 SUITE ABORTED ***
[error] Error during tests:
[error] org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite
[error] (catalyst / Test / testOnly) sbt.TestsFailedException: Tests
unsuccessful
[error] Total time: 66 s (01:06), completed Jan 27, 2025, 9:52:35 AM
For the record, we keep it in the `master` branch still.
Dongjoon.
On Mon, Jan 27, 2025 at 12:33 PM Martin Grund <ma...@databricks.com.invalid>
wrote:
> Would it not have been mindful to wait for the original author to
> investigate the PR and do a forward fix instead of reverting such a big
> change? Since this was only blocking the Maven test we could have waited
> probably a few more days without any issues.
>
> On Mon, Jan 27, 2025 at 8:32 PM Dongjoon Hyun <do...@apache.org> wrote:
>
>> This is reverted from branch-4.0 via the following.
>>
>> - https://github.com/apache/spark/pull/49696
>> Revert "[SPARK-49700][CONNECT][SQL] Unified Scala Interface for Connect
>> and Classic"
>>
>> Dongjoon.
>>
>> On 2025/01/26 16:58:45 Dongjoon Hyun wrote:
>> > Thank you!
>> >
>> > Dongjoon
>> >
>> > On Sat, Jan 25, 2025 at 20:01 Yang Jie <ya...@apache.org> wrote:
>> >
>> > > I reported a test issue that is suspected to be related to this pr：
>> > >
>> > > - https://github.com/apache/spark/pull/48818/files#r1929652392
>> > >
>> > > and it seems to be causing the failure of the Maven daily test.
>> > >
>> > > Thanks,
>> > > Jie Yang
>> > >
>> > > On 2025/01/24 20:24:57 Dongjoon Hyun wrote:
>> > > > Hi, All.
>> > > >
>> > > > SPARK-49700 landed one hour ago.
>> > > >
>> > > > Since this is another huge package redesign across 399 files in
>> Spark
>> > > 4.0,
>> > > > please check if you are not affected accidentally.
>> > > >
>> > > > Best Regards,
>> > > > Dongjoon.
>> > > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> > >
>> > >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: FYI: SPARK-49700 Unified Scala Interface for Connect and Classic
Posted by Martin Grund <ma...@databricks.com.INVALID>.
Would it not have been mindful to wait for the original author to
investigate the PR and do a forward fix instead of reverting such a big
change? Since this was only blocking the Maven test we could have waited
probably a few more days without any issues.
On Mon, Jan 27, 2025 at 8:32 PM Dongjoon Hyun <do...@apache.org> wrote:
> This is reverted from branch-4.0 via the following.
>
> - https://github.com/apache/spark/pull/49696
> Revert "[SPARK-49700][CONNECT][SQL] Unified Scala Interface for Connect
> and Classic"
>
> Dongjoon.
>
> On 2025/01/26 16:58:45 Dongjoon Hyun wrote:
> > Thank you!
> >
> > Dongjoon
> >
> > On Sat, Jan 25, 2025 at 20:01 Yang Jie <ya...@apache.org> wrote:
> >
> > > I reported a test issue that is suspected to be related to this pr：
> > >
> > > - https://github.com/apache/spark/pull/48818/files#r1929652392
> > >
> > > and it seems to be causing the failure of the Maven daily test.
> > >
> > > Thanks,
> > > Jie Yang
> > >
> > > On 2025/01/24 20:24:57 Dongjoon Hyun wrote:
> > > > Hi, All.
> > > >
> > > > SPARK-49700 landed one hour ago.
> > > >
> > > > Since this is another huge package redesign across 399 files in Spark
> > > 4.0,
> > > > please check if you are not affected accidentally.
> > > >
> > > > Best Regards,
> > > > Dongjoon.
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: FYI: SPARK-49700 Unified Scala Interface for Connect and Classic
Posted by Dongjoon Hyun <do...@apache.org>.
This is reverted from branch-4.0 via the following.
- https://github.com/apache/spark/pull/49696
Revert "[SPARK-49700][CONNECT][SQL] Unified Scala Interface for Connect and Classic"
Dongjoon.
On 2025/01/26 16:58:45 Dongjoon Hyun wrote:
> Thank you!
>
> Dongjoon
>
> On Sat, Jan 25, 2025 at 20:01 Yang Jie <ya...@apache.org> wrote:
>
> > I reported a test issue that is suspected to be related to this pr：
> >
> > - https://github.com/apache/spark/pull/48818/files#r1929652392
> >
> > and it seems to be causing the failure of the Maven daily test.
> >
> > Thanks,
> > Jie Yang
> >
> > On 2025/01/24 20:24:57 Dongjoon Hyun wrote:
> > > Hi, All.
> > >
> > > SPARK-49700 landed one hour ago.
> > >
> > > Since this is another huge package redesign across 399 files in Spark
> > 4.0,
> > > please check if you are not affected accidentally.
> > >
> > > Best Regards,
> > > Dongjoon.
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: FYI: SPARK-49700 Unified Scala Interface for Connect and Classic
Posted by Dongjoon Hyun <do...@gmail.com>.
Thank you!
Dongjoon
On Sat, Jan 25, 2025 at 20:01 Yang Jie <ya...@apache.org> wrote:
> I reported a test issue that is suspected to be related to this pr：
>
> - https://github.com/apache/spark/pull/48818/files#r1929652392
>
> and it seems to be causing the failure of the Maven daily test.
>
> Thanks,
> Jie Yang
>
> On 2025/01/24 20:24:57 Dongjoon Hyun wrote:
> > Hi, All.
> >
> > SPARK-49700 landed one hour ago.
> >
> > Since this is another huge package redesign across 399 files in Spark
> 4.0,
> > please check if you are not affected accidentally.
> >
> > Best Regards,
> > Dongjoon.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: FYI: SPARK-49700 Unified Scala Interface for Connect and Classic
Posted by Yang Jie <ya...@apache.org>.
I reported a test issue that is suspected to be related to this pr：
- https://github.com/apache/spark/pull/48818/files#r1929652392
and it seems to be causing the failure of the Maven daily test.
Thanks,
Jie Yang
On 2025/01/24 20:24:57 Dongjoon Hyun wrote:
> Hi, All.
>
> SPARK-49700 landed one hour ago.
>
> Since this is another huge package redesign across 399 files in Spark 4.0,
> please check if you are not affected accidentally.
>
> Best Regards,
> Dongjoon.
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org