[jira] [Comment Edited] (IGNITE-12054) Upgrade Spark module to 2.4

["Nikolay Izhikov (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IGNITE-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927325#comment-16927325 ] 

Nikolay Izhikov edited comment on IGNITE-12054 at 9/11/19 7:34 AM:
-------------------------------------------------------------------

Hello, [~zaleslaw]

The main issue with upgrading to 2.4 and support both 2.3 and 2.5 is changes in Spark internal API that were introduced in 2.3 version.
As you may know, Ignite - Spark integration uses some internal API ({{ExternalCatalog}}) and leverage on other parts of close-to-internal parts of Spark(query parser and optimizer). That code was changed in 2.4 and we can't simply change the version in pom.xml. We need to fix our code for the new Spark behaviour.

I tried to do so but fail to do it quickly.
If you want to take care of this upgrade, please, do.

As for support both version, it seems we should have different modules or similar to do so.

[~dmagda] Ignite, in the past, had those kinds of modules, that support two different versions of some external library.
Should we do the similar and create spark_24 and spark_25 modules with the very similar(but still different) codebase?


was (Author: nizhikov):
Hello, [~zaleslaw]

The main issue with upgrading to 2.4 and support both 2.3 and 2.5 is changes in Spark internal API that was introduced in 2.3 version.
As you may know, Ignite - Spark integration uses some internal API ({{ExternalCatalog}}) and leverage on other parts of close-to-internal parts of Spark(query parser and optimizer). That code was changed in 2.4 and we can't simply change the version in pom.xml. We need to fix our code for the new Spark behaviour.

I tried to do so but fail to do it quickly.
If you want to take care of this upgrade, please, do.

As for support both version, it seems we should have different modules or similar to do so.

[~dmagda] Ignite, in the past, had those kinds of modules, that support two different versions of some external library.
Should we do the similar and create spark_24 and spark_25 modules with the very similar(but still different) codebase?

> Upgrade Spark module to 2.4
> ---------------------------
>
>                 Key: IGNITE-12054
>                 URL: https://issues.apache.org/jira/browse/IGNITE-12054
>             Project: Ignite
>          Issue Type: Task
>          Components: spark
>    Affects Versions: 2.7.5
>            Reporter: Denis Magda
>            Assignee: Nikolay Izhikov
>            Priority: Blocker
>             Fix For: 2.8
>
>
> Users can't use APIs that are already available in Spark 2.4:
> https://stackoverflow.com/questions/57392143/persisting-spark-dataframe-to-ignite
> Let's upgrade Spark from 2.3 to 2.4 until we extract the Spark Integration as a separate module that can support multiple Spark versions.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)