You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Alex Ott <al...@gmail.com> on 2021/02/08 18:01:21 UTC
Binary compatibility issues in 3.1.1?
Hi all
I've noticed following SO question about Spark 3.1.1 not working with Delta
0.7.0:
https://stackoverflow.com/questions/66106096/delta-lake-insert-into-sql-in-pyspark-is-failing-with-java-lang-nosuchmethoder/66106800#66106800
- I checked with Delta 0.8.0 and it has the same problem. It brokes here:
https://github.com/delta-io/delta/blob/v0.8.0/src/main/scala/org/apache/spark/sql/delta/DeltaAnalysis.scala#L204
I'm thinking that it's caused by following change:
https://github.com/apache/spark/commit/a082f4600b1cb814442beed1b578bc3430a257a7#diff-cf96171d13fd77e670764766ae22afafbc4a396316bd758a89b60a6fe70d5b0dL150,
but not 100% sure. If it's, then maybe we could add the backward
compatible constructor for this case class?
P.S. It's not the first time when 3.1 breaks compatibility with existing
connectors, for example, Spark Cassandra Connector on 3.1 doesn't work
without changes:
https://github.com/datastax/spark-cassandra-connector/pull/1280
--
With best wishes, Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)
Re: Binary compatibility issues in 3.1.1?
Posted by Wenchen Fan <cl...@gmail.com>.
This is the cost of relying on Spark internal APIs, and the external
connectors need to take care of it.
BTW, the Alias change is source-compatible, and it shouldn't break the
external connectors if they are compiled with Spark 3.1.
On Tue, Feb 9, 2021 at 2:26 AM Alex Ott <al...@gmail.com> wrote:
> although no, additional constructor won't work...
>
> On Mon, Feb 8, 2021 at 7:01 PM Alex Ott <al...@gmail.com> wrote:
>
>> Hi all
>>
>> I've noticed following SO question about Spark 3.1.1 not working with
>> Delta 0.7.0:
>> https://stackoverflow.com/questions/66106096/delta-lake-insert-into-sql-in-pyspark-is-failing-with-java-lang-nosuchmethoder/66106800#66106800
>> - I checked with Delta 0.8.0 and it has the same problem. It brokes here:
>> https://github.com/delta-io/delta/blob/v0.8.0/src/main/scala/org/apache/spark/sql/delta/DeltaAnalysis.scala#L204
>>
>> I'm thinking that it's caused by following change:
>> https://github.com/apache/spark/commit/a082f4600b1cb814442beed1b578bc3430a257a7#diff-cf96171d13fd77e670764766ae22afafbc4a396316bd758a89b60a6fe70d5b0dL150,
>> but not 100% sure. If it's, then maybe we could add the backward
>> compatible constructor for this case class?
>>
>> P.S. It's not the first time when 3.1 breaks compatibility with existing
>> connectors, for example, Spark Cassandra Connector on 3.1 doesn't work
>> without changes:
>> https://github.com/datastax/spark-cassandra-connector/pull/1280
>>
>>
>> --
>> With best wishes, Alex Ott
>> http://alexott.net/
>> Twitter: alexott_en (English), alexott (Russian)
>>
>
>
> --
> With best wishes, Alex Ott
> http://alexott.net/
> Twitter: alexott_en (English), alexott (Russian)
>
Re: Binary compatibility issues in 3.1.1?
Posted by Alex Ott <al...@gmail.com>.
although no, additional constructor won't work...
On Mon, Feb 8, 2021 at 7:01 PM Alex Ott <al...@gmail.com> wrote:
> Hi all
>
> I've noticed following SO question about Spark 3.1.1 not working with
> Delta 0.7.0:
> https://stackoverflow.com/questions/66106096/delta-lake-insert-into-sql-in-pyspark-is-failing-with-java-lang-nosuchmethoder/66106800#66106800
> - I checked with Delta 0.8.0 and it has the same problem. It brokes here:
> https://github.com/delta-io/delta/blob/v0.8.0/src/main/scala/org/apache/spark/sql/delta/DeltaAnalysis.scala#L204
>
> I'm thinking that it's caused by following change:
> https://github.com/apache/spark/commit/a082f4600b1cb814442beed1b578bc3430a257a7#diff-cf96171d13fd77e670764766ae22afafbc4a396316bd758a89b60a6fe70d5b0dL150,
> but not 100% sure. If it's, then maybe we could add the backward
> compatible constructor for this case class?
>
> P.S. It's not the first time when 3.1 breaks compatibility with existing
> connectors, for example, Spark Cassandra Connector on 3.1 doesn't work
> without changes:
> https://github.com/datastax/spark-cassandra-connector/pull/1280
>
>
> --
> With best wishes, Alex Ott
> http://alexott.net/
> Twitter: alexott_en (English), alexott (Russian)
>
--
With best wishes, Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)