You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@ignite.apache.org by Николай Ижиков <ni...@gmail.com> on 2017/11/24 10:55:34 UTC

TC issues. IGNITE-3084. Spark Data Frame API

Hello, guys.

I have some issues on TC with my PR [1] for IGNITE-3084(Spark Data Frame API).
Can you, please, help me:


1. `Ignite RDD spark 2_10` -

Currently this build runs with following profiles: `-Plgpl,examples,scala-2.10,-clean-libs,-release` [2]
That means `scala` profile is activated too for `Ignite RDD spark 2_10`
Because `scala` activation is done like [3]:

```
             <activation>
                 <property><name>!scala-2.10</name></property>
             </activation>
```

I think it a misconfiguration because scala(2.11) shouldn't be activated for 2.10 build.
Am I miss something?

Can someone edit build property?
	* Add `-scala` to profiles list
	* Or add `-Dscala-2.10` to jvm properties to turn off `scala` profile in this build.


2. `Ignite RDD` -

Currently this build run on jvm7 [4].
As I wrote in my previous mail [5] current version of spark(2.2) runs only on jvm8.

Can someone edit build property to run it on jvm8?


3. For now `Ignite RDD` and `Ignite RDD spark 2_10` only runs java tests [6] existing in `spark` module.
There are several existing tests written in scala(i.e. scala-test) ignored in TC. IgniteRDDSpec [7] for example.
Is it turned off by a purpose or I miss something?
Should we run scala-test for spark and spark_2.10 modules?


[1] https://github.com/apache/ignite/pull/2742
[2] https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
[3] https://github.com/apache/ignite/blob/master/pom.xml#L533
[4] https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
[5] http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
[6] https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
[7] https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala

Re: TC issues. IGNITE-3084. Spark Data Frame API

Posted by Николай Ижиков <ni...@gmail.com>.

Valentin,

Now it's run OK.

Thank you.

30.11.2017 23:41, Valentin Kulichenko пишет:
> Nikolay,
> 
> Please try once again.
> 
> -Val
> 
> On Thu, Nov 30, 2017 at 11:43 AM, Николай Ижиков <nizhikov.dev@gmail.com <ma...@gmail.com>> wrote:
> 
>     Valentin,
> 
>     Thank you, but your changes does not enough
> 
>     I ran a build for my branch and still has "Unsupport major.minor version 52.0" issue [1]
> 
>     Build log:
> 
>     `Starting: /usr/lib/jvm/java-7-oracle/bin/java -DJAVA_HOME=/usr/lib/jvm/java-8-oracle`
> 
>     I look to build settings and found some variables that sill points to jdk7:
> 
>     Environment variables:
> 
>     env.JAVA_HOME   /usr/lib/jvm/java-7-oracle
>     env.JDK_HOME    /usr/lib/jvm/java-7-oracle
> 
>     Can you, please, change this variables too?
> 
>     [1] https://ci.ignite.apache.org/viewLog.html?buildId=970913&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildLog
>     <https://ci.ignite.apache.org/viewLog.html?buildId=970913&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildLog>
> 
> 
>     30.11.2017 22:16, Valentin Kulichenko пишет:
> 
>         Nikolay,
> 
>         Java 7 support will be dropped by Ignite soon, so let's do the upgrade now. I changed both 'Ignite RDD' and 'Ignite RDD spark 2_10' configuration on TC to use JDK 8. Can you try it out and let
>         me know if it works?
> 
>         -Val
> 
>         On Wed, Nov 29, 2017 at 11:28 PM, Николай Ижиков <nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>> wrote:
> 
>              Valentin,
> 
>              > Oh, so this is because of upgrade to 2.2.0?
> 
>              Yes, we should upgrade spark module to jdk1.8 because switching to spark 2.2.0.
> 
>              > least we should consider not dropping previous version yet.
> 
>              Please, note, we can have IgniteRDD and Ignite Data Frame for a spark 2.1.
> 
>              The only thing we can't have is IgniteCatalog.
> 
>              Currently, in my PR I include IgniteRDD and Ignite Data Frame in spark_2.10 module
>              So we doesn't have to drop spark 2.1 completely.
> 
>         https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742> <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>>
> 
>              30.11.2017 02:09, Valentin Kulichenko пишет:
> 
>                  Nikolay,
> 
>                  Oh, so this is because of upgrade to 2.2.0? Then I'm not sure we should upgrade in the first place, or at least we should consider not dropping previous version yet. I sent a message
>         to the
>                  original thread about upgrade, let's decide there and then come back to this issue.
> 
>                  -Val
> 
>                  On Tue, Nov 28, 2017 at 8:08 PM, Николай Ижиков <nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>
>         <mailto:nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>>> wrote:
> 
>                       Valentin,
> 
>                       For now `Ignite RDD` build runs on jdk1.7.
>                       We need to update it to jdk1.8.
> 
>                       I wrote the whole versions numbers to be clear:
> 
>                       1. Current master - Spark version is 2.1.0.
>                                So both `Ignite RDD` and `Ignite RDD 2.10` runs OK on jdk1.7.
> 
>                       2. My branch -
>                                `Ignite RDD 2.10` - spark version is 2.1.2 - runs OK on jdk1.7.
>                                `Ignite RDD` - spark version 2.2.0 - fails on jdk1.7, *has to be changed to run on jdk1.8*
> 
> 
>                       29.11.2017 03:27, Valentin Kulichenko пишет:
> 
>                           Nikolay,
> 
>                           If Spark requires Java 8, then I guess we have no choice. How TC is configured at the moment? My understanding is that Spark related suites are successfully executed there, so is
>                  there an issue?
> 
>                           -Val
> 
>                           On Tue, Nov 28, 2017 at 2:42 AM, Николай Ижиков <nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>
>         <mailto:nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>>
>                  <mailto:nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>> <mailto:nizhikov.dev@gmail.com
>         <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>>>> wrote:
> 
>                                Hello, Valentin.
> 
>                                    Added '-Dscala-2.10' to the build config. Let me know if it helps.
> 
> 
>                                Yes, it helps. Thank you!
>                                Now, 'Ignite RDD spark 2_10' succeed for my branch.
> 
> 
>                                    Do you mean that IgniteRDD does not compile on JDK7? If yes, do we know the reason? I don't think switching it to JDK8 is a solution as it should work with both.
> 
> 
>                                I mean that latest version of spark doesn't support jdk7.
> 
>         http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/> <http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/>> <http://spark.apache.org/docs/latest/
>         <http://spark.apache.org/docs/latest/> <http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/>>> <http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/>
>                  <http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/>> <http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/>
>         <http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/>>>>
> 
>                                "Spark runs on Java 8+..."
>                                "For the Scala API, Spark 2.2.0 uses Scala 2.11..."
>                                "Note that support for Java 7... were removed as of Spark 2.2.0"
>                                "Note that support for Scala 2.10 is deprecated..."
> 
>                                Moreover, We can't have IgniteCatalog for spark 2.1.
>                                Please, see my explanation in jira ticket -
> 
>         https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523
>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>
>                  <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523
>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>>
>                           <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523
>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>
>                  <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523
>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>>>
>                                <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523
>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>
>                  <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523
>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>>
>                           <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523
>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>
>                  <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523
>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>>>>
> 
>                                Do you see any options to support jdk7 for spark module?
> 
>                                > I think all tests should be executed on TC. Can you check if they work and add them to corresponding suites
> 
>                                OK, I file a ticket and try to fix it shortly.
> 
>         https://issues.apache.org/jira/browse/IGNITE-7042 <https://issues.apache.org/jira/browse/IGNITE-7042> <https://issues.apache.org/jira/browse/IGNITE-7042
>         <https://issues.apache.org/jira/browse/IGNITE-7042>> <https://issues.apache.org/jira/browse/IGNITE-7042 <https://issues.apache.org/jira/browse/IGNITE-7042>
>                  <https://issues.apache.org/jira/browse/IGNITE-7042 <https://issues.apache.org/jira/browse/IGNITE-7042>>> <https://issues.apache.org/jira/browse/IGNITE-7042
>         <https://issues.apache.org/jira/browse/IGNITE-7042> <https://issues.apache.org/jira/browse/IGNITE-7042 <https://issues.apache.org/jira/browse/IGNITE-7042>>
> 
>                           <https://issues.apache.org/jira/browse/IGNITE-7042 <https://issues.apache.org/jira/browse/IGNITE-7042> <https://issues.apache.org/jira/browse/IGNITE-7042
>         <https://issues.apache.org/jira/browse/IGNITE-7042>>>>
> 
>                                28.11.2017 03:33, Valentin Kulichenko пишет:
> 
>                                    Hi Nikolay,
> 
>                                    Please see my responses inline.
> 
>                                    -Val
> 
>                                    On Fri, Nov 24, 2017 at 2:55 AM, Николай Ижиков <nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com
>         <ma...@gmail.com>> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>>
>                  <mailto:nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>> <mailto:nizhikov.dev@gmail.com
>         <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>>>
>                           <mailto:nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>> <mailto:nizhikov.dev@gmail.com
>         <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>
>                  <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com
>         <ma...@gmail.com>>>>>> wrote:
> 
>                                         Hello, guys.
> 
>                                         I have some issues on TC with my PR [1] for IGNITE-3084(Spark Data Frame API).
>                                         Can you, please, help me:
> 
> 
>                                         1. `Ignite RDD spark 2_10` -
> 
>                                         Currently this build runs with following profiles: `-Plgpl,examples,scala-2.10,-clean-libs,-release` [2]
>                                         That means `scala` profile is activated too for `Ignite RDD spark 2_10`
>                                         Because `scala` activation is done like [3]:
> 
>                                         ```
>                                                      <activation>
>                                                          <property><name>!scala-2.10</name></property>
>                                                      </activation>
>                                         ```
> 
>                                         I think it a misconfiguration because scala(2.11) shouldn't be activated for 2.10 build.
>                                         Am I miss something?
> 
>                                         Can someone edit build property?
>                                                  * Add `-scala` to profiles list
>                                                  * Or add `-Dscala-2.10` to jvm properties to turn off `scala` profile in this build.
> 
> 
>                                    Added '-Dscala-2.10' to the build config. Let me know if it helps.
> 
> 
>                                         2. `Ignite RDD` -
> 
>                                         Currently this build run on jvm7 [4].
>                                         As I wrote in my previous mail [5] current version of spark(2.2) runs only on jvm8.
> 
>                                         Can someone edit build property to run it on jvm8?
> 
> 
>                                    Do you mean that IgniteRDD does not compile on JDK7? If yes, do we know the reason? I don't think switching it to JDK8 is a solution as it should work with both.
> 
> 
>                                         3. For now `Ignite RDD` and `Ignite RDD spark 2_10` only runs java tests [6] existing in `spark` module.
>                                         There are several existing tests written in scala(i.e. scala-test) ignored in TC. IgniteRDDSpec [7] for example.
>                                         Is it turned off by a purpose or I miss something?
>                                         Should we run scala-test for spark and spark_2.10 modules?
> 
>                                    I think all tests should be executed on TC. Can you check if they work and add them to corresponding suites?
> 
> 
>                                         [1] https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742> <https://github.com/apache/ignite/pull/2742
>         <https://github.com/apache/ignite/pull/2742>> <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>
>                  <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>>> <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>
>         <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>>
>                           <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742> <https://github.com/apache/ignite/pull/2742
>         <https://github.com/apache/ignite/pull/2742>>>> <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742> <https://github.com/apache/ignite/pull/2742
>         <https://github.com/apache/ignite/pull/2742>>
>                  <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742> <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>>>
>         <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742> <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>>
>                           <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742> <https://github.com/apache/ignite/pull/2742
>         <https://github.com/apache/ignite/pull/2742>>>>>
>                                         [2] https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>>>
>                                    <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>>>>
>                                         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>>>
>                                    <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>>>>>
>                                         [3] https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>
>         <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>>
>                  <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533> <https://github.com/apache/ignite/blob/master/pom.xml#L533
>         <https://github.com/apache/ignite/blob/master/pom.xml#L533>>>
>                           <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>
>         <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>> <https://github.com/apache/ignite/blob/master/pom.xml#L533
>         <https://github.com/apache/ignite/blob/master/pom.xml#L533>
>                  <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>>>> <https://github.com/apache/ignite/blob/master/pom.xml#L533
>         <https://github.com/apache/ignite/blob/master/pom.xml#L533> <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>>
> 
> 
>                           <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>
>         <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>>>
>                                    <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>
>         <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>>
>                  <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533> <https://github.com/apache/ignite/blob/master/pom.xml#L533
>         <https://github.com/apache/ignite/blob/master/pom.xml#L533>>>>>
>                                         [4] https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>>
>                                    <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>>>
>                                         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>>
>                                    <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>>>>
>                                         [5] http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>                  <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>
>                           <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>                  <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>>
>                                    <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>                  <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>
>                           <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>                  <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>>>
>                                         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>                  <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>
>                           <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>                  <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>>
>                                    <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>                  <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>
>                           <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>                  <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>>>>
>                                         [6] https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>>
>                                    <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>>>
>                                         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>>
>                                    <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>>>>
>                                         [7] https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>                  <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>
>                           <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>                  <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>>
>                                    <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>                  <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>
>                           <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>                  <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>>>
>                                         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>                  <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>
>                           <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>                  <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>>
>                                    <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>                  <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>
>                           <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>                  <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>>>>
> 
> 
> 
> 
> 
>

Re: TC issues. IGNITE-3084. Spark Data Frame API

Posted by Valentin Kulichenko <va...@gmail.com>.

Nikolay,

Please try once again.

-Val

On Thu, Nov 30, 2017 at 11:43 AM, Николай Ижиков <ni...@gmail.com>
wrote:

> Valentin,
>
> Thank you, but your changes does not enough
>
> I ran a build for my branch and still has "Unsupport major.minor version
> 52.0" issue [1]
>
> Build log:
>
> `Starting: /usr/lib/jvm/java-7-oracle/bin/java
> -DJAVA_HOME=/usr/lib/jvm/java-8-oracle`
>
> I look to build settings and found some variables that sill points to jdk7:
>
> Environment variables:
>
> env.JAVA_HOME   /usr/lib/jvm/java-7-oracle
> env.JDK_HOME    /usr/lib/jvm/java-7-oracle
>
> Can you, please, change this variables too?
>
> [1] https://ci.ignite.apache.org/viewLog.html?buildId=970913&bui
> ldTypeId=Ignite20Tests_IgniteRdd&tab=buildLog
>
>
> 30.11.2017 22:16, Valentin Kulichenko пишет:
>
>> Nikolay,
>>
>> Java 7 support will be dropped by Ignite soon, so let's do the upgrade
>> now. I changed both 'Ignite RDD' and 'Ignite RDD spark 2_10' configuration
>> on TC to use JDK 8. Can you try it out and let me know if it works?
>>
>> -Val
>>
>> On Wed, Nov 29, 2017 at 11:28 PM, Николай Ижиков <nizhikov.dev@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>>     Valentin,
>>
>>     > Oh, so this is because of upgrade to 2.2.0?
>>
>>     Yes, we should upgrade spark module to jdk1.8 because switching to
>> spark 2.2.0.
>>
>>     > least we should consider not dropping previous version yet.
>>
>>     Please, note, we can have IgniteRDD and Ignite Data Frame for a spark
>> 2.1.
>>
>>     The only thing we can't have is IgniteCatalog.
>>
>>     Currently, in my PR I include IgniteRDD and Ignite Data Frame in
>> spark_2.10 module
>>     So we doesn't have to drop spark 2.1 completely.
>>
>>     https://github.com/apache/ignite/pull/2742 <
>> https://github.com/apache/ignite/pull/2742>
>>
>>     30.11.2017 02:09, Valentin Kulichenko пишет:
>>
>>         Nikolay,
>>
>>         Oh, so this is because of upgrade to 2.2.0? Then I'm not sure we
>> should upgrade in the first place, or at least we should consider not
>> dropping previous version yet. I sent a message to the
>>         original thread about upgrade, let's decide there and then come
>> back to this issue.
>>
>>         -Val
>>
>>         On Tue, Nov 28, 2017 at 8:08 PM, Николай Ижиков <
>> nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:
>> nizhikov.dev@gmail.com <ma...@gmail.com>>> wrote:
>>
>>              Valentin,
>>
>>              For now `Ignite RDD` build runs on jdk1.7.
>>              We need to update it to jdk1.8.
>>
>>              I wrote the whole versions numbers to be clear:
>>
>>              1. Current master - Spark version is 2.1.0.
>>                       So both `Ignite RDD` and `Ignite RDD 2.10` runs OK
>> on jdk1.7.
>>
>>              2. My branch -
>>                       `Ignite RDD 2.10` - spark version is 2.1.2 - runs
>> OK on jdk1.7.
>>                       `Ignite RDD` - spark version 2.2.0 - fails on
>> jdk1.7, *has to be changed to run on jdk1.8*
>>
>>
>>              29.11.2017 03:27, Valentin Kulichenko пишет:
>>
>>                  Nikolay,
>>
>>                  If Spark requires Java 8, then I guess we have no
>> choice. How TC is configured at the moment? My understanding is that Spark
>> related suites are successfully executed there, so is
>>         there an issue?
>>
>>                  -Val
>>
>>                  On Tue, Nov 28, 2017 at 2:42 AM, Николай Ижиков <
>> nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:
>> nizhikov.dev@gmail.com <ma...@gmail.com>>
>>         <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>
>> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>>> wrote:
>>
>>                       Hello, Valentin.
>>
>>                           Added '-Dscala-2.10' to the build config. Let
>> me know if it helps.
>>
>>
>>                       Yes, it helps. Thank you!
>>                       Now, 'Ignite RDD spark 2_10' succeed for my branch.
>>
>>
>>                           Do you mean that IgniteRDD does not compile on
>> JDK7? If yes, do we know the reason? I don't think switching it to JDK8 is
>> a solution as it should work with both.
>>
>>
>>                       I mean that latest version of spark doesn't support
>> jdk7.
>>
>>         http://spark.apache.org/docs/latest/ <
>> http://spark.apache.org/docs/latest/> <http://spark.apache.org/docs/
>> latest/ <http://spark.apache.org/docs/latest/>> <
>> http://spark.apache.org/docs/latest/
>>         <http://spark.apache.org/docs/latest/> <
>> http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/
>> latest/>>>
>>
>>                       "Spark runs on Java 8+..."
>>                       "For the Scala API, Spark 2.2.0 uses Scala 2.11..."
>>                       "Note that support for Java 7... were removed as of
>> Spark 2.2.0"
>>                       "Note that support for Scala 2.10 is deprecated..."
>>
>>                       Moreover, We can't have IgniteCatalog for spark 2.1.
>>                       Please, see my explanation in jira ticket -
>>
>>         https://issues.apache.org/jira/browse/IGNITE-3084?focusedCom
>> mentId=16268523&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-16268523
>>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCo
>> mmentId=16268523&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-16268523>
>>                  <https://issues.apache.org/jir
>> a/browse/IGNITE-3084?focusedCommentId=16268523&page=com.
>> atlassian.jira.plugin.system.issuetabpanels:comment-
>> tabpanel#comment-16268523
>>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCo
>> mmentId=16268523&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-16268523>>
>>                       <https://issues.apache.org/jir
>> a/browse/IGNITE-3084?focusedCommentId=16268523&page=com.
>> atlassian.jira.plugin.system.issuetabpanels:comment-
>> tabpanel#comment-16268523
>>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCo
>> mmentId=16268523&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-16268523>
>>                  <https://issues.apache.org/jir
>> a/browse/IGNITE-3084?focusedCommentId=16268523&page=com.
>> atlassian.jira.plugin.system.issuetabpanels:comment-
>> tabpanel#comment-16268523
>>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCo
>> mmentId=16268523&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-16268523>>>
>>
>>                       Do you see any options to support jdk7 for spark
>> module?
>>
>>                       > I think all tests should be executed on TC. Can
>> you check if they work and add them to corresponding suites
>>
>>                       OK, I file a ticket and try to fix it shortly.
>>
>>         https://issues.apache.org/jira/browse/IGNITE-7042 <
>> https://issues.apache.org/jira/browse/IGNITE-7042> <
>> https://issues.apache.org/jira/browse/IGNITE-7042
>>         <https://issues.apache.org/jira/browse/IGNITE-7042>> <
>> https://issues.apache.org/jira/browse/IGNITE-7042 <
>> https://issues.apache.org/jira/browse/IGNITE-7042>
>>
>>                  <https://issues.apache.org/jira/browse/IGNITE-7042 <
>> https://issues.apache.org/jira/browse/IGNITE-7042>>>
>>
>>                       28.11.2017 03:33, Valentin Kulichenko пишет:
>>
>>                           Hi Nikolay,
>>
>>                           Please see my responses inline.
>>
>>                           -Val
>>
>>                           On Fri, Nov 24, 2017 at 2:55 AM, Николай Ижиков
>> <nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:
>> nizhikov.dev@gmail.com <ma...@gmail.com>>
>>         <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>
>> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>>
>>                  <mailto:nizhikov.dev@gmail.com <mailto:
>> nizhikov.dev@gmail.com> <mailto:nizhikov.dev@gmail.com <mailto:
>> nizhikov.dev@gmail.com>> <mailto:nizhikov.dev@gmail.com
>>         <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com
>> <ma...@gmail.com>>>>> wrote:
>>
>>                                Hello, guys.
>>
>>                                I have some issues on TC with my PR [1]
>> for IGNITE-3084(Spark Data Frame API).
>>                                Can you, please, help me:
>>
>>
>>                                1. `Ignite RDD spark 2_10` -
>>
>>                                Currently this build runs with following
>> profiles: `-Plgpl,examples,scala-2.10,-clean-libs,-release` [2]
>>                                That means `scala` profile is activated
>> too for `Ignite RDD spark 2_10`
>>                                Because `scala` activation is done like
>> [3]:
>>
>>                                ```
>>                                             <activation>
>>
>> <property><name>!scala-2.10</name></property>
>>                                             </activation>
>>                                ```
>>
>>                                I think it a misconfiguration because
>> scala(2.11) shouldn't be activated for 2.10 build.
>>                                Am I miss something?
>>
>>                                Can someone edit build property?
>>                                         * Add `-scala` to profiles list
>>                                         * Or add `-Dscala-2.10` to jvm
>> properties to turn off `scala` profile in this build.
>>
>>
>>                           Added '-Dscala-2.10' to the build config. Let
>> me know if it helps.
>>
>>
>>                                2. `Ignite RDD` -
>>
>>                                Currently this build run on jvm7 [4].
>>                                As I wrote in my previous mail [5] current
>> version of spark(2.2) runs only on jvm8.
>>
>>                                Can someone edit build property to run it
>> on jvm8?
>>
>>
>>                           Do you mean that IgniteRDD does not compile on
>> JDK7? If yes, do we know the reason? I don't think switching it to JDK8 is
>> a solution as it should work with both.
>>
>>
>>                                3. For now `Ignite RDD` and `Ignite RDD
>> spark 2_10` only runs java tests [6] existing in `spark` module.
>>                                There are several existing tests written
>> in scala(i.e. scala-test) ignored in TC. IgniteRDDSpec [7] for example.
>>                                Is it turned off by a purpose or I miss
>> something?
>>                                Should we run scala-test for spark and
>> spark_2.10 modules?
>>
>>                           I think all tests should be executed on TC. Can
>> you check if they work and add them to corresponding suites?
>>
>>
>>                                [1] https://github.com/apache/igni
>> te/pull/2742 <https://github.com/apache/ignite/pull/2742> <
>> https://github.com/apache/ignite/pull/2742
>>         <https://github.com/apache/ignite/pull/2742>> <
>> https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ign
>> ite/pull/2742>
>>                  <https://github.com/apache/ignite/pull/2742 <
>> https://github.com/apache/ignite/pull/2742>>> <
>> https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ign
>> ite/pull/2742>
>>         <https://github.com/apache/ignite/pull/2742 <
>> https://github.com/apache/ignite/pull/2742>> <
>> https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ign
>> ite/pull/2742>
>>                  <https://github.com/apache/ignite/pull/2742 <
>> https://github.com/apache/ignite/pull/2742>>>>
>>                                [2] https://ci.ignite.apache.org/v
>> iewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteR
>> ddSpark210&tab=buildLog&_focus=379#_state=371
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
>> focus=379#_state=371>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=buildLog&_focus=379#_state=371
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
>> focus=379#_state=371>>
>>                           <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=buildLog&_focus=379#_state=371
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
>> focus=379#_state=371>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=buildLog&_focus=379#_state=371
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
>> focus=379#_state=371>>>
>>                                <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=buildLog&_focus=379#_state=371
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
>> focus=379#_state=371>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=buildLog&_focus=379#_state=371
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
>> focus=379#_state=371>>
>>                           <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=buildLog&_focus=379#_state=371
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
>> focus=379#_state=371>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=buildLog&_focus=379#_state=371
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
>> focus=379#_state=371>>>>
>>                                [3] https://github.com/apache/igni
>> te/blob/master/pom.xml#L533 <https://github.com/apache/ign
>> ite/blob/master/pom.xml#L533>
>>         <https://github.com/apache/ignite/blob/master/pom.xml#L533 <
>> https://github.com/apache/ignite/blob/master/pom.xml#L533>>
>>                  <https://github.com/apache/ign
>> ite/blob/master/pom.xml#L533 <https://github.com/apache/ign
>> ite/blob/master/pom.xml#L533> <https://github.com/apache/ign
>> ite/blob/master/pom.xml#L533
>>         <https://github.com/apache/ignite/blob/master/pom.xml#L533>>> <
>> https://github.com/apache/ignite/blob/master/pom.xml#L533 <
>> https://github.com/apache/ignite/blob/master/pom.xml#L533>
>>
>>
>>                  <https://github.com/apache/ign
>> ite/blob/master/pom.xml#L533 <https://github.com/apache/ign
>> ite/blob/master/pom.xml#L533>>
>>                           <https://github.com/apache/ign
>> ite/blob/master/pom.xml#L533 <https://github.com/apache/ign
>> ite/blob/master/pom.xml#L533>
>>         <https://github.com/apache/ignite/blob/master/pom.xml#L533 <
>> https://github.com/apache/ignite/blob/master/pom.xml#L533>>>>
>>                                [4] https://ci.ignite.apache.org/v
>> iewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteR
>> dd&tab=buildParameters
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_Ignite
>> Rdd&tab=buildParameters
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>
>>                           <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_Ignite
>> Rdd&tab=buildParameters
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_Ignite
>> Rdd&tab=buildParameters
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>>
>>                                <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_Ignite
>> Rdd&tab=buildParameters
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_Ignite
>> Rdd&tab=buildParameters
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>
>>                           <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_Ignite
>> Rdd&tab=buildParameters
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_Ignite
>> Rdd&tab=buildParameters
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>>>
>>                                [5] http://apache-ignite-developer
>> s.2346864.n4.nabble.com/Integration-of-Spark-and-Ignit
>> e-Prototype-tp22649p23099.html
>>         <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>>                  <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>>         <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>
>>                           <http://apache-ignite-develope
>> rs.2346864.n4.nabble.com/Integration-of-Spark-and-Ignit
>> e-Prototype-tp22649p23099.html
>>         <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>>                  <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>>         <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>>
>>                                <http://apache-ignite-develope
>> rs.2346864.n4.nabble.com/Integration-of-Spark-and-Ignit
>> e-Prototype-tp22649p23099.html
>>         <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>>                  <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>>         <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>
>>                           <http://apache-ignite-develope
>> rs.2346864.n4.nabble.com/Integration-of-Spark-and-Ignit
>> e-Prototype-tp22649p23099.html
>>         <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>>                  <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>>         <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>>>
>>                                [6] https://ci.ignite.apache.org/v
>> iewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteR
>> ddSpark210&tab=testsInfo
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=testsInfo
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>
>>                           <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=testsInfo
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=testsInfo
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>>
>>                                <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=testsInfo
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=testsInfo
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>
>>                           <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=testsInfo
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=testsInfo
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>>>
>>                                [7] https://github.com/apache/igni
>> te/blob/master/modules/spark/src/test/scala/org/apache/
>> ignite/spark/IgniteRDDSpec.scala
>>         <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>>                  <https://github.com/apache/ign
>> ite/blob/master/modules/spark/src/test/scala/org/apache/
>> ignite/spark/IgniteRDDSpec.scala
>>         <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>
>>                           <https://github.com/apache/ign
>> ite/blob/master/modules/spark/src/test/scala/org/apache/
>> ignite/spark/IgniteRDDSpec.scala
>>         <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>>                  <https://github.com/apache/ign
>> ite/blob/master/modules/spark/src/test/scala/org/apache/
>> ignite/spark/IgniteRDDSpec.scala
>>         <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>>
>>                                <https://github.com/apache/ign
>> ite/blob/master/modules/spark/src/test/scala/org/apache/
>> ignite/spark/IgniteRDDSpec.scala
>>         <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>>                  <https://github.com/apache/ign
>> ite/blob/master/modules/spark/src/test/scala/org/apache/
>> ignite/spark/IgniteRDDSpec.scala
>>         <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>
>>                           <https://github.com/apache/ign
>> ite/blob/master/modules/spark/src/test/scala/org/apache/
>> ignite/spark/IgniteRDDSpec.scala
>>         <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>>                  <https://github.com/apache/ign
>> ite/blob/master/modules/spark/src/test/scala/org/apache/
>> ignite/spark/IgniteRDDSpec.scala
>>         <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>>>
>>
>>
>>
>>
>>
>>

Re: TC issues. IGNITE-3084. Spark Data Frame API

Posted by Николай Ижиков <ni...@gmail.com>.

Valentin,

Thank you, but your changes does not enough

I ran a build for my branch and still has "Unsupport major.minor version 52.0" issue [1]

Build log:

`Starting: /usr/lib/jvm/java-7-oracle/bin/java -DJAVA_HOME=/usr/lib/jvm/java-8-oracle`

I look to build settings and found some variables that sill points to jdk7:

Environment variables:

env.JAVA_HOME	/usr/lib/jvm/java-7-oracle
env.JDK_HOME	/usr/lib/jvm/java-7-oracle

Can you, please, change this variables too?

[1] https://ci.ignite.apache.org/viewLog.html?buildId=970913&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildLog


30.11.2017 22:16, Valentin Kulichenko пишет:
> Nikolay,
> 
> Java 7 support will be dropped by Ignite soon, so let's do the upgrade now. I changed both 'Ignite RDD' and 'Ignite RDD spark 2_10' configuration on TC to use JDK 8. Can you try it out and let me know 
> if it works?
> 
> -Val
> 
> On Wed, Nov 29, 2017 at 11:28 PM, Николай Ижиков <nizhikov.dev@gmail.com <ma...@gmail.com>> wrote:
> 
>     Valentin,
> 
>     > Oh, so this is because of upgrade to 2.2.0?
> 
>     Yes, we should upgrade spark module to jdk1.8 because switching to spark 2.2.0.
> 
>     > least we should consider not dropping previous version yet.
> 
>     Please, note, we can have IgniteRDD and Ignite Data Frame for a spark 2.1.
> 
>     The only thing we can't have is IgniteCatalog.
> 
>     Currently, in my PR I include IgniteRDD and Ignite Data Frame in spark_2.10 module
>     So we doesn't have to drop spark 2.1 completely.
> 
>     https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>
> 
>     30.11.2017 02:09, Valentin Kulichenko пишет:
> 
>         Nikolay,
> 
>         Oh, so this is because of upgrade to 2.2.0? Then I'm not sure we should upgrade in the first place, or at least we should consider not dropping previous version yet. I sent a message to the
>         original thread about upgrade, let's decide there and then come back to this issue.
> 
>         -Val
> 
>         On Tue, Nov 28, 2017 at 8:08 PM, Николай Ижиков <nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>> wrote:
> 
>              Valentin,
> 
>              For now `Ignite RDD` build runs on jdk1.7.
>              We need to update it to jdk1.8.
> 
>              I wrote the whole versions numbers to be clear:
> 
>              1. Current master - Spark version is 2.1.0.
>                       So both `Ignite RDD` and `Ignite RDD 2.10` runs OK on jdk1.7.
> 
>              2. My branch -
>                       `Ignite RDD 2.10` - spark version is 2.1.2 - runs OK on jdk1.7.
>                       `Ignite RDD` - spark version 2.2.0 - fails on jdk1.7, *has to be changed to run on jdk1.8*
> 
> 
>              29.11.2017 03:27, Valentin Kulichenko пишет:
> 
>                  Nikolay,
> 
>                  If Spark requires Java 8, then I guess we have no choice. How TC is configured at the moment? My understanding is that Spark related suites are successfully executed there, so is
>         there an issue?
> 
>                  -Val
> 
>                  On Tue, Nov 28, 2017 at 2:42 AM, Николай Ижиков <nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>
>         <mailto:nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>>> wrote:
> 
>                       Hello, Valentin.
> 
>                           Added '-Dscala-2.10' to the build config. Let me know if it helps.
> 
> 
>                       Yes, it helps. Thank you!
>                       Now, 'Ignite RDD spark 2_10' succeed for my branch.
> 
> 
>                           Do you mean that IgniteRDD does not compile on JDK7? If yes, do we know the reason? I don't think switching it to JDK8 is a solution as it should work with both.
> 
> 
>                       I mean that latest version of spark doesn't support jdk7.
> 
>         http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/> <http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/>> <http://spark.apache.org/docs/latest/
>         <http://spark.apache.org/docs/latest/> <http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/>>>
> 
>                       "Spark runs on Java 8+..."
>                       "For the Scala API, Spark 2.2.0 uses Scala 2.11..."
>                       "Note that support for Java 7... were removed as of Spark 2.2.0"
>                       "Note that support for Scala 2.10 is deprecated..."
> 
>                       Moreover, We can't have IgniteCatalog for spark 2.1.
>                       Please, see my explanation in jira ticket -
> 
>         https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523
>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>
>                  <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523
>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>>
>                       <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523
>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>
>                  <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523
>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>>>
> 
>                       Do you see any options to support jdk7 for spark module?
> 
>                       > I think all tests should be executed on TC. Can you check if they work and add them to corresponding suites
> 
>                       OK, I file a ticket and try to fix it shortly.
> 
>         https://issues.apache.org/jira/browse/IGNITE-7042 <https://issues.apache.org/jira/browse/IGNITE-7042> <https://issues.apache.org/jira/browse/IGNITE-7042
>         <https://issues.apache.org/jira/browse/IGNITE-7042>> <https://issues.apache.org/jira/browse/IGNITE-7042 <https://issues.apache.org/jira/browse/IGNITE-7042>
>                  <https://issues.apache.org/jira/browse/IGNITE-7042 <https://issues.apache.org/jira/browse/IGNITE-7042>>>
> 
>                       28.11.2017 03:33, Valentin Kulichenko пишет:
> 
>                           Hi Nikolay,
> 
>                           Please see my responses inline.
> 
>                           -Val
> 
>                           On Fri, Nov 24, 2017 at 2:55 AM, Николай Ижиков <nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>
>         <mailto:nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>>
>                  <mailto:nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>> <mailto:nizhikov.dev@gmail.com
>         <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>>>> wrote:
> 
>                                Hello, guys.
> 
>                                I have some issues on TC with my PR [1] for IGNITE-3084(Spark Data Frame API).
>                                Can you, please, help me:
> 
> 
>                                1. `Ignite RDD spark 2_10` -
> 
>                                Currently this build runs with following profiles: `-Plgpl,examples,scala-2.10,-clean-libs,-release` [2]
>                                That means `scala` profile is activated too for `Ignite RDD spark 2_10`
>                                Because `scala` activation is done like [3]:
> 
>                                ```
>                                             <activation>
>                                                 <property><name>!scala-2.10</name></property>
>                                             </activation>
>                                ```
> 
>                                I think it a misconfiguration because scala(2.11) shouldn't be activated for 2.10 build.
>                                Am I miss something?
> 
>                                Can someone edit build property?
>                                         * Add `-scala` to profiles list
>                                         * Or add `-Dscala-2.10` to jvm properties to turn off `scala` profile in this build.
> 
> 
>                           Added '-Dscala-2.10' to the build config. Let me know if it helps.
> 
> 
>                                2. `Ignite RDD` -
> 
>                                Currently this build run on jvm7 [4].
>                                As I wrote in my previous mail [5] current version of spark(2.2) runs only on jvm8.
> 
>                                Can someone edit build property to run it on jvm8?
> 
> 
>                           Do you mean that IgniteRDD does not compile on JDK7? If yes, do we know the reason? I don't think switching it to JDK8 is a solution as it should work with both.
> 
> 
>                                3. For now `Ignite RDD` and `Ignite RDD spark 2_10` only runs java tests [6] existing in `spark` module.
>                                There are several existing tests written in scala(i.e. scala-test) ignored in TC. IgniteRDDSpec [7] for example.
>                                Is it turned off by a purpose or I miss something?
>                                Should we run scala-test for spark and spark_2.10 modules?
> 
>                           I think all tests should be executed on TC. Can you check if they work and add them to corresponding suites?
> 
> 
>                                [1] https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742> <https://github.com/apache/ignite/pull/2742
>         <https://github.com/apache/ignite/pull/2742>> <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>
>                  <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>>> <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>
>         <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>> <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>
>                  <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>>>>
>                                [2] https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>>>
>                                <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>>>>
>                                [3] https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>
>         <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>>
>                  <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533> <https://github.com/apache/ignite/blob/master/pom.xml#L533
>         <https://github.com/apache/ignite/blob/master/pom.xml#L533>>> <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>
> 
>                  <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>>
>                           <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>
>         <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>>>>
>                                [4] https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>>
>                                <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>>>
>                                [5] http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>                  <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>
>                           <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>                  <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>>
>                                <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>                  <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>
>                           <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>                  <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>>>
>                                [6] https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>>
>                                <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>
>                           <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>>>
>                                [7] https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>                  <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>
>                           <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>                  <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>>
>                                <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>                  <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>
>                           <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>                  <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>>>
> 
> 
> 
> 
>

Re: TC issues. IGNITE-3084. Spark Data Frame API

Posted by Valentin Kulichenko <va...@gmail.com>.

Nikolay,

Java 7 support will be dropped by Ignite soon, so let's do the upgrade now.
I changed both 'Ignite RDD' and 'Ignite RDD spark 2_10' configuration on TC
to use JDK 8. Can you try it out and let me know if it works?

-Val

On Wed, Nov 29, 2017 at 11:28 PM, Николай Ижиков <ni...@gmail.com>
wrote:

> Valentin,
>
> > Oh, so this is because of upgrade to 2.2.0?
>
> Yes, we should upgrade spark module to jdk1.8 because switching to spark
> 2.2.0.
>
> > least we should consider not dropping previous version yet.
>
> Please, note, we can have IgniteRDD and Ignite Data Frame for a spark 2.1.
>
> The only thing we can't have is IgniteCatalog.
>
> Currently, in my PR I include IgniteRDD and Ignite Data Frame in
> spark_2.10 module
> So we doesn't have to drop spark 2.1 completely.
>
> https://github.com/apache/ignite/pull/2742
>
> 30.11.2017 02:09, Valentin Kulichenko пишет:
>
>> Nikolay,
>>
>> Oh, so this is because of upgrade to 2.2.0? Then I'm not sure we should
>> upgrade in the first place, or at least we should consider not dropping
>> previous version yet. I sent a message to the original thread about
>> upgrade, let's decide there and then come back to this issue.
>>
>> -Val
>>
>> On Tue, Nov 28, 2017 at 8:08 PM, Николай Ижиков <nizhikov.dev@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>>     Valentin,
>>
>>     For now `Ignite RDD` build runs on jdk1.7.
>>     We need to update it to jdk1.8.
>>
>>     I wrote the whole versions numbers to be clear:
>>
>>     1. Current master - Spark version is 2.1.0.
>>              So both `Ignite RDD` and `Ignite RDD 2.10` runs OK on jdk1.7.
>>
>>     2. My branch -
>>              `Ignite RDD 2.10` - spark version is 2.1.2 - runs OK on
>> jdk1.7.
>>              `Ignite RDD` - spark version 2.2.0 - fails on jdk1.7, *has
>> to be changed to run on jdk1.8*
>>
>>
>>     29.11.2017 03:27, Valentin Kulichenko пишет:
>>
>>         Nikolay,
>>
>>         If Spark requires Java 8, then I guess we have no choice. How TC
>> is configured at the moment? My understanding is that Spark related suites
>> are successfully executed there, so is there an issue?
>>
>>         -Val
>>
>>         On Tue, Nov 28, 2017 at 2:42 AM, Николай Ижиков <
>> nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:
>> nizhikov.dev@gmail.com <ma...@gmail.com>>> wrote:
>>
>>              Hello, Valentin.
>>
>>                  Added '-Dscala-2.10' to the build config. Let me know if
>> it helps.
>>
>>
>>              Yes, it helps. Thank you!
>>              Now, 'Ignite RDD spark 2_10' succeed for my branch.
>>
>>
>>                  Do you mean that IgniteRDD does not compile on JDK7? If
>> yes, do we know the reason? I don't think switching it to JDK8 is a
>> solution as it should work with both.
>>
>>
>>              I mean that latest version of spark doesn't support jdk7.
>>
>>         http://spark.apache.org/docs/latest/ <
>> http://spark.apache.org/docs/latest/> <http://spark.apache.org/docs/
>> latest/ <http://spark.apache.org/docs/latest/>>
>>
>>              "Spark runs on Java 8+..."
>>              "For the Scala API, Spark 2.2.0 uses Scala 2.11..."
>>              "Note that support for Java 7... were removed as of Spark
>> 2.2.0"
>>              "Note that support for Scala 2.10 is deprecated..."
>>
>>              Moreover, We can't have IgniteCatalog for spark 2.1.
>>              Please, see my explanation in jira ticket -
>>
>>         https://issues.apache.org/jira/browse/IGNITE-3084?focusedCom
>> mentId=16268523&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-16268523
>>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCo
>> mmentId=16268523&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-16268523>
>>              <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCo
>> mmentId=16268523&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-16268523
>>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCo
>> mmentId=16268523&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-16268523>>
>>
>>              Do you see any options to support jdk7 for spark module?
>>
>>              > I think all tests should be executed on TC. Can you check
>> if they work and add them to corresponding suites
>>
>>              OK, I file a ticket and try to fix it shortly.
>>
>>         https://issues.apache.org/jira/browse/IGNITE-7042 <
>> https://issues.apache.org/jira/browse/IGNITE-7042> <
>> https://issues.apache.org/jira/browse/IGNITE-7042
>>         <https://issues.apache.org/jira/browse/IGNITE-7042>>
>>
>>              28.11.2017 03:33, Valentin Kulichenko пишет:
>>
>>                  Hi Nikolay,
>>
>>                  Please see my responses inline.
>>
>>                  -Val
>>
>>                  On Fri, Nov 24, 2017 at 2:55 AM, Николай Ижиков <
>> nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:
>> nizhikov.dev@gmail.com <ma...@gmail.com>>
>>         <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>
>> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>>> wrote:
>>
>>                       Hello, guys.
>>
>>                       I have some issues on TC with my PR [1] for
>> IGNITE-3084(Spark Data Frame API).
>>                       Can you, please, help me:
>>
>>
>>                       1. `Ignite RDD spark 2_10` -
>>
>>                       Currently this build runs with following profiles:
>> `-Plgpl,examples,scala-2.10,-clean-libs,-release` [2]
>>                       That means `scala` profile is activated too for
>> `Ignite RDD spark 2_10`
>>                       Because `scala` activation is done like [3]:
>>
>>                       ```
>>                                    <activation>
>>                                        <property><name>!scala-2.10</n
>> ame></property>
>>                                    </activation>
>>                       ```
>>
>>                       I think it a misconfiguration because scala(2.11)
>> shouldn't be activated for 2.10 build.
>>                       Am I miss something?
>>
>>                       Can someone edit build property?
>>                                * Add `-scala` to profiles list
>>                                * Or add `-Dscala-2.10` to jvm properties
>> to turn off `scala` profile in this build.
>>
>>
>>                  Added '-Dscala-2.10' to the build config. Let me know if
>> it helps.
>>
>>
>>                       2. `Ignite RDD` -
>>
>>                       Currently this build run on jvm7 [4].
>>                       As I wrote in my previous mail [5] current version
>> of spark(2.2) runs only on jvm8.
>>
>>                       Can someone edit build property to run it on jvm8?
>>
>>
>>                  Do you mean that IgniteRDD does not compile on JDK7? If
>> yes, do we know the reason? I don't think switching it to JDK8 is a
>> solution as it should work with both.
>>
>>
>>                       3. For now `Ignite RDD` and `Ignite RDD spark 2_10`
>> only runs java tests [6] existing in `spark` module.
>>                       There are several existing tests written in
>> scala(i.e. scala-test) ignored in TC. IgniteRDDSpec [7] for example.
>>                       Is it turned off by a purpose or I miss something?
>>                       Should we run scala-test for spark and spark_2.10
>> modules?
>>
>>                  I think all tests should be executed on TC. Can you
>> check if they work and add them to corresponding suites?
>>
>>
>>                       [1] https://github.com/apache/ignite/pull/2742 <
>> https://github.com/apache/ignite/pull/2742> <
>> https://github.com/apache/ignite/pull/2742
>>         <https://github.com/apache/ignite/pull/2742>> <
>> https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ign
>> ite/pull/2742> <https://github.com/apache/ignite/pull/2742
>>         <https://github.com/apache/ignite/pull/2742>>>
>>                       [2] https://ci.ignite.apache.org/v
>> iewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteR
>> ddSpark210&tab=buildLog&_focus=379#_state=371
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
>> focus=379#_state=371>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=buildLog&_focus=379#_state=371
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
>> focus=379#_state=371>>
>>                       <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=buildLog&_focus=379#_state=371
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
>> focus=379#_state=371>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=buildLog&_focus=379#_state=371
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
>> focus=379#_state=371>>>
>>                       [3] https://github.com/apache/igni
>> te/blob/master/pom.xml#L533 <https://github.com/apache/ign
>> ite/blob/master/pom.xml#L533>
>>         <https://github.com/apache/ignite/blob/master/pom.xml#L533 <
>> https://github.com/apache/ignite/blob/master/pom.xml#L533>> <
>> https://github.com/apache/ignite/blob/master/pom.xml#L533
>>
>>         <https://github.com/apache/ignite/blob/master/pom.xml#L533>
>>                  <https://github.com/apache/ign
>> ite/blob/master/pom.xml#L533 <https://github.com/apache/ign
>> ite/blob/master/pom.xml#L533>>>
>>                       [4] https://ci.ignite.apache.org/v
>> iewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteR
>> dd&tab=buildParameters
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_Ignite
>> Rdd&tab=buildParameters
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>
>>                       <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_Ignite
>> Rdd&tab=buildParameters
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_Ignite
>> Rdd&tab=buildParameters
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>>
>>                       [5] http://apache-ignite-developer
>> s.2346864.n4.nabble.com/Integration-of-Spark-and-Ignit
>> e-Prototype-tp22649p23099.html
>>         <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>>                  <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>>         <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>
>>                       <http://apache-ignite-develope
>> rs.2346864.n4.nabble.com/Integration-of-Spark-and-Ignit
>> e-Prototype-tp22649p23099.html
>>         <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>>                  <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>>         <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>>
>>                       [6] https://ci.ignite.apache.org/v
>> iewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteR
>> ddSpark210&tab=testsInfo
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=testsInfo
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>
>>                       <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=testsInfo
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>>                  <https://ci.ignite.apache.org/
>> viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_Ignite
>> RddSpark210&tab=testsInfo
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>>
>>                       [7] https://github.com/apache/igni
>> te/blob/master/modules/spark/src/test/scala/org/apache/
>> ignite/spark/IgniteRDDSpec.scala
>>         <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>>                  <https://github.com/apache/ign
>> ite/blob/master/modules/spark/src/test/scala/org/apache/
>> ignite/spark/IgniteRDDSpec.scala
>>         <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>
>>                       <https://github.com/apache/ign
>> ite/blob/master/modules/spark/src/test/scala/org/apache/
>> ignite/spark/IgniteRDDSpec.scala
>>         <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>>                  <https://github.com/apache/ign
>> ite/blob/master/modules/spark/src/test/scala/org/apache/
>> ignite/spark/IgniteRDDSpec.scala
>>         <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>>
>>
>>
>>
>>
>>

Re: TC issues. IGNITE-3084. Spark Data Frame API

Posted by Николай Ижиков <ni...@gmail.com>.

Valentin,

 > Oh, so this is because of upgrade to 2.2.0?

Yes, we should upgrade spark module to jdk1.8 because switching to spark 2.2.0.

 > least we should consider not dropping previous version yet.

Please, note, we can have IgniteRDD and Ignite Data Frame for a spark 2.1.

The only thing we can't have is IgniteCatalog.

Currently, in my PR I include IgniteRDD and Ignite Data Frame in spark_2.10 module
So we doesn't have to drop spark 2.1 completely.

https://github.com/apache/ignite/pull/2742

30.11.2017 02:09, Valentin Kulichenko пишет:
> Nikolay,
> 
> Oh, so this is because of upgrade to 2.2.0? Then I'm not sure we should upgrade in the first place, or at least we should consider not dropping previous version yet. I sent a message to the original 
> thread about upgrade, let's decide there and then come back to this issue.
> 
> -Val
> 
> On Tue, Nov 28, 2017 at 8:08 PM, Николай Ижиков <nizhikov.dev@gmail.com <ma...@gmail.com>> wrote:
> 
>     Valentin,
> 
>     For now `Ignite RDD` build runs on jdk1.7.
>     We need to update it to jdk1.8.
> 
>     I wrote the whole versions numbers to be clear:
> 
>     1. Current master - Spark version is 2.1.0.
>              So both `Ignite RDD` and `Ignite RDD 2.10` runs OK on jdk1.7.
> 
>     2. My branch -
>              `Ignite RDD 2.10` - spark version is 2.1.2 - runs OK on jdk1.7.
>              `Ignite RDD` - spark version 2.2.0 - fails on jdk1.7, *has to be changed to run on jdk1.8*
> 
> 
>     29.11.2017 03:27, Valentin Kulichenko пишет:
> 
>         Nikolay,
> 
>         If Spark requires Java 8, then I guess we have no choice. How TC is configured at the moment? My understanding is that Spark related suites are successfully executed there, so is there an issue?
> 
>         -Val
> 
>         On Tue, Nov 28, 2017 at 2:42 AM, Николай Ижиков <nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>> wrote:
> 
>              Hello, Valentin.
> 
>                  Added '-Dscala-2.10' to the build config. Let me know if it helps.
> 
> 
>              Yes, it helps. Thank you!
>              Now, 'Ignite RDD spark 2_10' succeed for my branch.
> 
> 
>                  Do you mean that IgniteRDD does not compile on JDK7? If yes, do we know the reason? I don't think switching it to JDK8 is a solution as it should work with both.
> 
> 
>              I mean that latest version of spark doesn't support jdk7.
> 
>         http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/> <http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/>>
> 
>              "Spark runs on Java 8+..."
>              "For the Scala API, Spark 2.2.0 uses Scala 2.11..."
>              "Note that support for Java 7... were removed as of Spark 2.2.0"
>              "Note that support for Scala 2.10 is deprecated..."
> 
>              Moreover, We can't have IgniteCatalog for spark 2.1.
>              Please, see my explanation in jira ticket -
> 
>         https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523
>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>
>              <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523
>         <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>>
> 
>              Do you see any options to support jdk7 for spark module?
> 
>              > I think all tests should be executed on TC. Can you check if they work and add them to corresponding suites
> 
>              OK, I file a ticket and try to fix it shortly.
> 
>         https://issues.apache.org/jira/browse/IGNITE-7042 <https://issues.apache.org/jira/browse/IGNITE-7042> <https://issues.apache.org/jira/browse/IGNITE-7042
>         <https://issues.apache.org/jira/browse/IGNITE-7042>>
> 
>              28.11.2017 03:33, Valentin Kulichenko пишет:
> 
>                  Hi Nikolay,
> 
>                  Please see my responses inline.
> 
>                  -Val
> 
>                  On Fri, Nov 24, 2017 at 2:55 AM, Николай Ижиков <nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>
>         <mailto:nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>>> wrote:
> 
>                       Hello, guys.
> 
>                       I have some issues on TC with my PR [1] for IGNITE-3084(Spark Data Frame API).
>                       Can you, please, help me:
> 
> 
>                       1. `Ignite RDD spark 2_10` -
> 
>                       Currently this build runs with following profiles: `-Plgpl,examples,scala-2.10,-clean-libs,-release` [2]
>                       That means `scala` profile is activated too for `Ignite RDD spark 2_10`
>                       Because `scala` activation is done like [3]:
> 
>                       ```
>                                    <activation>
>                                        <property><name>!scala-2.10</name></property>
>                                    </activation>
>                       ```
> 
>                       I think it a misconfiguration because scala(2.11) shouldn't be activated for 2.10 build.
>                       Am I miss something?
> 
>                       Can someone edit build property?
>                                * Add `-scala` to profiles list
>                                * Or add `-Dscala-2.10` to jvm properties to turn off `scala` profile in this build.
> 
> 
>                  Added '-Dscala-2.10' to the build config. Let me know if it helps.
> 
> 
>                       2. `Ignite RDD` -
> 
>                       Currently this build run on jvm7 [4].
>                       As I wrote in my previous mail [5] current version of spark(2.2) runs only on jvm8.
> 
>                       Can someone edit build property to run it on jvm8?
> 
> 
>                  Do you mean that IgniteRDD does not compile on JDK7? If yes, do we know the reason? I don't think switching it to JDK8 is a solution as it should work with both.
> 
> 
>                       3. For now `Ignite RDD` and `Ignite RDD spark 2_10` only runs java tests [6] existing in `spark` module.
>                       There are several existing tests written in scala(i.e. scala-test) ignored in TC. IgniteRDDSpec [7] for example.
>                       Is it turned off by a purpose or I miss something?
>                       Should we run scala-test for spark and spark_2.10 modules?
> 
>                  I think all tests should be executed on TC. Can you check if they work and add them to corresponding suites?
> 
> 
>                       [1] https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742> <https://github.com/apache/ignite/pull/2742
>         <https://github.com/apache/ignite/pull/2742>> <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742> <https://github.com/apache/ignite/pull/2742
>         <https://github.com/apache/ignite/pull/2742>>>
>                       [2] https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>>
>                       <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>>>
>                       [3] https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>
>         <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>> <https://github.com/apache/ignite/blob/master/pom.xml#L533
>         <https://github.com/apache/ignite/blob/master/pom.xml#L533>
>                  <https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>>>
>                       [4] https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>
>                       <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>>
>                       [5] http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>                  <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>
>                       <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>                  <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>>
>                       [6] https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>
>                       <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>                  <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>>
>                       [7] https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>                  <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>
>                       <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>                  <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>>
> 
> 
> 
>

Re: TC issues. IGNITE-3084. Spark Data Frame API

Posted by Valentin Kulichenko <va...@gmail.com>.

Nikolay,

Oh, so this is because of upgrade to 2.2.0? Then I'm not sure we should
upgrade in the first place, or at least we should consider not dropping
previous version yet. I sent a message to the original thread about
upgrade, let's decide there and then come back to this issue.

-Val

On Tue, Nov 28, 2017 at 8:08 PM, Николай Ижиков <ni...@gmail.com>
wrote:

> Valentin,
>
> For now `Ignite RDD` build runs on jdk1.7.
> We need to update it to jdk1.8.
>
> I wrote the whole versions numbers to be clear:
>
> 1. Current master - Spark version is 2.1.0.
>         So both `Ignite RDD` and `Ignite RDD 2.10` runs OK on jdk1.7.
>
> 2. My branch -
>         `Ignite RDD 2.10` - spark version is 2.1.2 - runs OK on jdk1.7.
>         `Ignite RDD` - spark version 2.2.0 - fails on jdk1.7, *has to be
> changed to run on jdk1.8*
>
>
> 29.11.2017 03:27, Valentin Kulichenko пишет:
>
>> Nikolay,
>>
>> If Spark requires Java 8, then I guess we have no choice. How TC is
>> configured at the moment? My understanding is that Spark related suites are
>> successfully executed there, so is there an issue?
>>
>> -Val
>>
>> On Tue, Nov 28, 2017 at 2:42 AM, Николай Ижиков <nizhikov.dev@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>>     Hello, Valentin.
>>
>>         Added '-Dscala-2.10' to the build config. Let me know if it helps.
>>
>>
>>     Yes, it helps. Thank you!
>>     Now, 'Ignite RDD spark 2_10' succeed for my branch.
>>
>>
>>         Do you mean that IgniteRDD does not compile on JDK7? If yes, do
>> we know the reason? I don't think switching it to JDK8 is a solution as it
>> should work with both.
>>
>>
>>     I mean that latest version of spark doesn't support jdk7.
>>
>>     http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/
>> latest/>
>>
>>     "Spark runs on Java 8+..."
>>     "For the Scala API, Spark 2.2.0 uses Scala 2.11..."
>>     "Note that support for Java 7... were removed as of Spark 2.2.0"
>>     "Note that support for Scala 2.10 is deprecated..."
>>
>>     Moreover, We can't have IgniteCatalog for spark 2.1.
>>     Please, see my explanation in jira ticket -
>>
>>     https://issues.apache.org/jira/browse/IGNITE-3084?focusedCom
>> mentId=16268523&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-16268523
>>     <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCo
>> mmentId=16268523&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-16268523>
>>
>>     Do you see any options to support jdk7 for spark module?
>>
>>     > I think all tests should be executed on TC. Can you check if they
>> work and add them to corresponding suites
>>
>>     OK, I file a ticket and try to fix it shortly.
>>
>>     https://issues.apache.org/jira/browse/IGNITE-7042 <
>> https://issues.apache.org/jira/browse/IGNITE-7042>
>>
>>     28.11.2017 03:33, Valentin Kulichenko пишет:
>>
>>         Hi Nikolay,
>>
>>         Please see my responses inline.
>>
>>         -Val
>>
>>         On Fri, Nov 24, 2017 at 2:55 AM, Николай Ижиков <
>> nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:
>> nizhikov.dev@gmail.com <ma...@gmail.com>>> wrote:
>>
>>              Hello, guys.
>>
>>              I have some issues on TC with my PR [1] for
>> IGNITE-3084(Spark Data Frame API).
>>              Can you, please, help me:
>>
>>
>>              1. `Ignite RDD spark 2_10` -
>>
>>              Currently this build runs with following profiles:
>> `-Plgpl,examples,scala-2.10,-clean-libs,-release` [2]
>>              That means `scala` profile is activated too for `Ignite RDD
>> spark 2_10`
>>              Because `scala` activation is done like [3]:
>>
>>              ```
>>                           <activation>
>>                               <property><name>!scala-2.10</n
>> ame></property>
>>                           </activation>
>>              ```
>>
>>              I think it a misconfiguration because scala(2.11) shouldn't
>> be activated for 2.10 build.
>>              Am I miss something?
>>
>>              Can someone edit build property?
>>                       * Add `-scala` to profiles list
>>                       * Or add `-Dscala-2.10` to jvm properties to turn
>> off `scala` profile in this build.
>>
>>
>>         Added '-Dscala-2.10' to the build config. Let me know if it helps.
>>
>>
>>              2. `Ignite RDD` -
>>
>>              Currently this build run on jvm7 [4].
>>              As I wrote in my previous mail [5] current version of
>> spark(2.2) runs only on jvm8.
>>
>>              Can someone edit build property to run it on jvm8?
>>
>>
>>         Do you mean that IgniteRDD does not compile on JDK7? If yes, do
>> we know the reason? I don't think switching it to JDK8 is a solution as it
>> should work with both.
>>
>>
>>              3. For now `Ignite RDD` and `Ignite RDD spark 2_10` only
>> runs java tests [6] existing in `spark` module.
>>              There are several existing tests written in scala(i.e.
>> scala-test) ignored in TC. IgniteRDDSpec [7] for example.
>>              Is it turned off by a purpose or I miss something?
>>              Should we run scala-test for spark and spark_2.10 modules?
>>
>>         I think all tests should be executed on TC. Can you check if they
>> work and add them to corresponding suites?
>>
>>
>>              [1] https://github.com/apache/ignite/pull/2742 <
>> https://github.com/apache/ignite/pull/2742> <
>> https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ign
>> ite/pull/2742>>
>>              [2] https://ci.ignite.apache.org/v
>> iewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteR
>> ddSpark210&tab=buildLog&_focus=379#_state=371
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
>> focus=379#_state=371>
>>              <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
>> focus=379#_state=371
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
>> focus=379#_state=371>>
>>              [3] https://github.com/apache/igni
>> te/blob/master/pom.xml#L533 <https://github.com/apache/ign
>> ite/blob/master/pom.xml#L533> <https://github.com/apache/ign
>> ite/blob/master/pom.xml#L533
>>         <https://github.com/apache/ignite/blob/master/pom.xml#L533>>
>>              [4] https://ci.ignite.apache.org/v
>> iewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteR
>> dd&tab=buildParameters
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>>              <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>
>>              [5] http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>>         <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>>              <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>>         <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>
>>              [6] https://ci.ignite.apache.org/v
>> iewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteR
>> ddSpark210&tab=testsInfo
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>>              <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>
>>              [7] https://github.com/apache/igni
>> te/blob/master/modules/spark/src/test/scala/org/apache/
>> ignite/spark/IgniteRDDSpec.scala
>>         <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>>              <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>>         <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>
>>
>>
>>
>>

Re: TC issues. IGNITE-3084. Spark Data Frame API

Posted by Николай Ижиков <ni...@gmail.com>.

Valentin,

For now `Ignite RDD` build runs on jdk1.7.
We need to update it to jdk1.8.

I wrote the whole versions numbers to be clear:

1. Current master - Spark version is 2.1.0.
	So both `Ignite RDD` and `Ignite RDD 2.10` runs OK on jdk1.7.

2. My branch -
	`Ignite RDD 2.10` - spark version is 2.1.2 - runs OK on jdk1.7.
	`Ignite RDD` - spark version 2.2.0 - fails on jdk1.7, *has to be changed to run on jdk1.8*


29.11.2017 03:27, Valentin Kulichenko пишет:
> Nikolay,
> 
> If Spark requires Java 8, then I guess we have no choice. How TC is configured at the moment? My understanding is that Spark related suites are successfully executed there, so is there an issue?
> 
> -Val
> 
> On Tue, Nov 28, 2017 at 2:42 AM, Николай Ижиков <nizhikov.dev@gmail.com <ma...@gmail.com>> wrote:
> 
>     Hello, Valentin.
> 
>         Added '-Dscala-2.10' to the build config. Let me know if it helps.
> 
> 
>     Yes, it helps. Thank you!
>     Now, 'Ignite RDD spark 2_10' succeed for my branch.
> 
> 
>         Do you mean that IgniteRDD does not compile on JDK7? If yes, do we know the reason? I don't think switching it to JDK8 is a solution as it should work with both.
> 
> 
>     I mean that latest version of spark doesn't support jdk7.
> 
>     http://spark.apache.org/docs/latest/ <http://spark.apache.org/docs/latest/>
> 
>     "Spark runs on Java 8+..."
>     "For the Scala API, Spark 2.2.0 uses Scala 2.11..."
>     "Note that support for Java 7... were removed as of Spark 2.2.0"
>     "Note that support for Scala 2.10 is deprecated..."
> 
>     Moreover, We can't have IgniteCatalog for spark 2.1.
>     Please, see my explanation in jira ticket -
> 
>     https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523
>     <https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523>
> 
>     Do you see any options to support jdk7 for spark module?
> 
>     > I think all tests should be executed on TC. Can you check if they work and add them to corresponding suites
> 
>     OK, I file a ticket and try to fix it shortly.
> 
>     https://issues.apache.org/jira/browse/IGNITE-7042 <https://issues.apache.org/jira/browse/IGNITE-7042>
> 
>     28.11.2017 03:33, Valentin Kulichenko пишет:
> 
>         Hi Nikolay,
> 
>         Please see my responses inline.
> 
>         -Val
> 
>         On Fri, Nov 24, 2017 at 2:55 AM, Николай Ижиков <nizhikov.dev@gmail.com <ma...@gmail.com> <mailto:nizhikov.dev@gmail.com <ma...@gmail.com>>> wrote:
> 
>              Hello, guys.
> 
>              I have some issues on TC with my PR [1] for IGNITE-3084(Spark Data Frame API).
>              Can you, please, help me:
> 
> 
>              1. `Ignite RDD spark 2_10` -
> 
>              Currently this build runs with following profiles: `-Plgpl,examples,scala-2.10,-clean-libs,-release` [2]
>              That means `scala` profile is activated too for `Ignite RDD spark 2_10`
>              Because `scala` activation is done like [3]:
> 
>              ```
>                           <activation>
>                               <property><name>!scala-2.10</name></property>
>                           </activation>
>              ```
> 
>              I think it a misconfiguration because scala(2.11) shouldn't be activated for 2.10 build.
>              Am I miss something?
> 
>              Can someone edit build property?
>                       * Add `-scala` to profiles list
>                       * Or add `-Dscala-2.10` to jvm properties to turn off `scala` profile in this build.
> 
> 
>         Added '-Dscala-2.10' to the build config. Let me know if it helps.
> 
> 
>              2. `Ignite RDD` -
> 
>              Currently this build run on jvm7 [4].
>              As I wrote in my previous mail [5] current version of spark(2.2) runs only on jvm8.
> 
>              Can someone edit build property to run it on jvm8?
> 
> 
>         Do you mean that IgniteRDD does not compile on JDK7? If yes, do we know the reason? I don't think switching it to JDK8 is a solution as it should work with both.
> 
> 
>              3. For now `Ignite RDD` and `Ignite RDD spark 2_10` only runs java tests [6] existing in `spark` module.
>              There are several existing tests written in scala(i.e. scala-test) ignored in TC. IgniteRDDSpec [7] for example.
>              Is it turned off by a purpose or I miss something?
>              Should we run scala-test for spark and spark_2.10 modules?
> 
>         I think all tests should be executed on TC. Can you check if they work and add them to corresponding suites?
> 
> 
>              [1] https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742> <https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>>
>              [2] https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>              <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>>
>              [3] https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533> <https://github.com/apache/ignite/blob/master/pom.xml#L533
>         <https://github.com/apache/ignite/blob/master/pom.xml#L533>>
>              [4] https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>              <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>>
>              [5] http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>              <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>         <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>>
>              [6] https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>              <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>         <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>>
>              [7] https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>              <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>         <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>>
> 
> 
>

Re: TC issues. IGNITE-3084. Spark Data Frame API

Posted by Valentin Kulichenko <va...@gmail.com>.

Nikolay,

If Spark requires Java 8, then I guess we have no choice. How TC is
configured at the moment? My understanding is that Spark related suites are
successfully executed there, so is there an issue?

-Val

On Tue, Nov 28, 2017 at 2:42 AM, Николай Ижиков <ni...@gmail.com>
wrote:

> Hello, Valentin.
>
> Added '-Dscala-2.10' to the build config. Let me know if it helps.
>>
>
> Yes, it helps. Thank you!
> Now, 'Ignite RDD spark 2_10' succeed for my branch.
>
>
> Do you mean that IgniteRDD does not compile on JDK7? If yes, do we know
>> the reason? I don't think switching it to JDK8 is a solution as it should
>> work with both.
>>
>
> I mean that latest version of spark doesn't support jdk7.
>
> http://spark.apache.org/docs/latest/
>
> "Spark runs on Java 8+..."
> "For the Scala API, Spark 2.2.0 uses Scala 2.11..."
> "Note that support for Java 7... were removed as of Spark 2.2.0"
> "Note that support for Scala 2.10 is deprecated..."
>
> Moreover, We can't have IgniteCatalog for spark 2.1.
> Please, see my explanation in jira ticket -
>
> https://issues.apache.org/jira/browse/IGNITE-3084?focusedCom
> mentId=16268523&page=com.atlassian.jira.plugin.system.issuet
> abpanels:comment-tabpanel#comment-16268523
>
> Do you see any options to support jdk7 for spark module?
>
> > I think all tests should be executed on TC. Can you check if they work
> and add them to corresponding suites
>
> OK, I file a ticket and try to fix it shortly.
>
> https://issues.apache.org/jira/browse/IGNITE-7042
>
> 28.11.2017 03:33, Valentin Kulichenko пишет:
>
>> Hi Nikolay,
>>
>> Please see my responses inline.
>>
>> -Val
>>
>> On Fri, Nov 24, 2017 at 2:55 AM, Николай Ижиков <nizhikov.dev@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>>     Hello, guys.
>>
>>     I have some issues on TC with my PR [1] for IGNITE-3084(Spark Data
>> Frame API).
>>     Can you, please, help me:
>>
>>
>>     1. `Ignite RDD spark 2_10` -
>>
>>     Currently this build runs with following profiles:
>> `-Plgpl,examples,scala-2.10,-clean-libs,-release` [2]
>>     That means `scala` profile is activated too for `Ignite RDD spark
>> 2_10`
>>     Because `scala` activation is done like [3]:
>>
>>     ```
>>                  <activation>
>>                      <property><name>!scala-2.10</name></property>
>>                  </activation>
>>     ```
>>
>>     I think it a misconfiguration because scala(2.11) shouldn't be
>> activated for 2.10 build.
>>     Am I miss something?
>>
>>     Can someone edit build property?
>>              * Add `-scala` to profiles list
>>              * Or add `-Dscala-2.10` to jvm properties to turn off
>> `scala` profile in this build.
>>
>>
>> Added '-Dscala-2.10' to the build config. Let me know if it helps.
>>
>>
>>     2. `Ignite RDD` -
>>
>>     Currently this build run on jvm7 [4].
>>     As I wrote in my previous mail [5] current version of spark(2.2) runs
>> only on jvm8.
>>
>>     Can someone edit build property to run it on jvm8?
>>
>>
>> Do you mean that IgniteRDD does not compile on JDK7? If yes, do we know
>> the reason? I don't think switching it to JDK8 is a solution as it should
>> work with both.
>>
>>
>>     3. For now `Ignite RDD` and `Ignite RDD spark 2_10` only runs java
>> tests [6] existing in `spark` module.
>>     There are several existing tests written in scala(i.e. scala-test)
>> ignored in TC. IgniteRDDSpec [7] for example.
>>     Is it turned off by a purpose or I miss something?
>>     Should we run scala-test for spark and spark_2.10 modules?
>>
>> I think all tests should be executed on TC. Can you check if they work
>> and add them to corresponding suites?
>>
>>
>>     [1] https://github.com/apache/ignite/pull/2742 <
>> https://github.com/apache/ignite/pull/2742>
>>     [2] https://ci.ignite.apache.org/viewLog.html?buildId=960220&bui
>> ldTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus
>> =379#_state=371
>>     <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focu
>> s=379#_state=371>
>>     [3] https://github.com/apache/ignite/blob/master/pom.xml#L533 <
>> https://github.com/apache/ignite/blob/master/pom.xml#L533>
>>     [4] https://ci.ignite.apache.org/viewLog.html?buildId=960221&bui
>> ldTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>>     <https://ci.ignite.apache.org/viewLog.html?buildId=960221&bu
>> ildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>>     [5] http://apache-ignite-developers.2346864.n4.nabble.com/Integr
>> ation-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>>     <http://apache-ignite-developers.2346864.n4.nabble.com/Integ
>> ration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>>     [6] https://ci.ignite.apache.org/viewLog.html?buildId=960220&bui
>> ldTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>>     <https://ci.ignite.apache.org/viewLog.html?buildId=960220&bu
>> ildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>>     [7] https://github.com/apache/ignite/blob/master/modules/spark/s
>> rc/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>>     <https://github.com/apache/ignite/blob/master/modules/spark/
>> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
>>
>>
>>

Re: TC issues. IGNITE-3084. Spark Data Frame API

Posted by Николай Ижиков <ni...@gmail.com>.

Hello, Valentin.

> Added '-Dscala-2.10' to the build config. Let me know if it helps.

Yes, it helps. Thank you!
Now, 'Ignite RDD spark 2_10' succeed for my branch.


> Do you mean that IgniteRDD does not compile on JDK7? If yes, do we know the reason? I don't think switching it to JDK8 is a solution as it should work with both.

I mean that latest version of spark doesn't support jdk7.

http://spark.apache.org/docs/latest/

"Spark runs on Java 8+..."
"For the Scala API, Spark 2.2.0 uses Scala 2.11..."
"Note that support for Java 7... were removed as of Spark 2.2.0"
"Note that support for Scala 2.10 is deprecated..."

Moreover, We can't have IgniteCatalog for spark 2.1.
Please, see my explanation in jira ticket -

https://issues.apache.org/jira/browse/IGNITE-3084?focusedCommentId=16268523&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16268523

Do you see any options to support jdk7 for spark module?

 > I think all tests should be executed on TC. Can you check if they work and add them to corresponding suites

OK, I file a ticket and try to fix it shortly.

https://issues.apache.org/jira/browse/IGNITE-7042

28.11.2017 03:33, Valentin Kulichenko пишет:
> Hi Nikolay,
> 
> Please see my responses inline.
> 
> -Val
> 
> On Fri, Nov 24, 2017 at 2:55 AM, Николай Ижиков <nizhikov.dev@gmail.com <ma...@gmail.com>> wrote:
> 
>     Hello, guys.
> 
>     I have some issues on TC with my PR [1] for IGNITE-3084(Spark Data Frame API).
>     Can you, please, help me:
> 
> 
>     1. `Ignite RDD spark 2_10` -
> 
>     Currently this build runs with following profiles: `-Plgpl,examples,scala-2.10,-clean-libs,-release` [2]
>     That means `scala` profile is activated too for `Ignite RDD spark 2_10`
>     Because `scala` activation is done like [3]:
> 
>     ```
>                  <activation>
>                      <property><name>!scala-2.10</name></property>
>                  </activation>
>     ```
> 
>     I think it a misconfiguration because scala(2.11) shouldn't be activated for 2.10 build.
>     Am I miss something?
> 
>     Can someone edit build property?
>              * Add `-scala` to profiles list
>              * Or add `-Dscala-2.10` to jvm properties to turn off `scala` profile in this build.
> 
> 
> Added '-Dscala-2.10' to the build config. Let me know if it helps.
> 
> 
>     2. `Ignite RDD` -
> 
>     Currently this build run on jvm7 [4].
>     As I wrote in my previous mail [5] current version of spark(2.2) runs only on jvm8.
> 
>     Can someone edit build property to run it on jvm8?
> 
> 
> Do you mean that IgniteRDD does not compile on JDK7? If yes, do we know the reason? I don't think switching it to JDK8 is a solution as it should work with both.
> 
> 
>     3. For now `Ignite RDD` and `Ignite RDD spark 2_10` only runs java tests [6] existing in `spark` module.
>     There are several existing tests written in scala(i.e. scala-test) ignored in TC. IgniteRDDSpec [7] for example.
>     Is it turned off by a purpose or I miss something?
>     Should we run scala-test for spark and spark_2.10 modules? 
> 
> 
> I think all tests should be executed on TC. Can you check if they work and add them to corresponding suites?
> 
> 
>     [1] https://github.com/apache/ignite/pull/2742 <https://github.com/apache/ignite/pull/2742>
>     [2] https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371
>     <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_focus=379#_state=371>
>     [3] https://github.com/apache/ignite/blob/master/pom.xml#L533 <https://github.com/apache/ignite/blob/master/pom.xml#L533>
>     [4] https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
>     <https://ci.ignite.apache.org/viewLog.html?buildId=960221&buildTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters>
>     [5] http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
>     <http://apache-ignite-developers.2346864.n4.nabble.com/Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html>
>     [6] https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
>     <https://ci.ignite.apache.org/viewLog.html?buildId=960220&buildTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo>
>     [7] https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>     <https://github.com/apache/ignite/blob/master/modules/spark/src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala>
> 
>

Re: TC issues. IGNITE-3084. Spark Data Frame API

Posted by Valentin Kulichenko <va...@gmail.com>.

Hi Nikolay,

Please see my responses inline.

-Val

On Fri, Nov 24, 2017 at 2:55 AM, Николай Ижиков <ni...@gmail.com>
wrote:

> Hello, guys.
>
> I have some issues on TC with my PR [1] for IGNITE-3084(Spark Data Frame
> API).
> Can you, please, help me:
>
>
> 1. `Ignite RDD spark 2_10` -
>
> Currently this build runs with following profiles:
> `-Plgpl,examples,scala-2.10,-clean-libs,-release` [2]
> That means `scala` profile is activated too for `Ignite RDD spark 2_10`
> Because `scala` activation is done like [3]:
>
> ```
>             <activation>
>                 <property><name>!scala-2.10</name></property>
>             </activation>
> ```
>
> I think it a misconfiguration because scala(2.11) shouldn't be activated
> for 2.10 build.
> Am I miss something?
>
> Can someone edit build property?
>         * Add `-scala` to profiles list
>         * Or add `-Dscala-2.10` to jvm properties to turn off `scala`
> profile in this build.
>
>
Added '-Dscala-2.10' to the build config. Let me know if it helps.


>
> 2. `Ignite RDD` -
>
> Currently this build run on jvm7 [4].
> As I wrote in my previous mail [5] current version of spark(2.2) runs only
> on jvm8.
>
> Can someone edit build property to run it on jvm8?
>
>
Do you mean that IgniteRDD does not compile on JDK7? If yes, do we know the
reason? I don't think switching it to JDK8 is a solution as it should work
with both.


>
> 3. For now `Ignite RDD` and `Ignite RDD spark 2_10` only runs java tests
> [6] existing in `spark` module.
> There are several existing tests written in scala(i.e. scala-test) ignored
> in TC. IgniteRDDSpec [7] for example.
> Is it turned off by a purpose or I miss something?
> Should we run scala-test for spark and spark_2.10 modules?


I think all tests should be executed on TC. Can you check if they work and
add them to corresponding suites?


>
> [1] https://github.com/apache/ignite/pull/2742
> [2] https://ci.ignite.apache.org/viewLog.html?buildId=960220&bui
> ldTypeId=Ignite20Tests_IgniteRddSpark210&tab=buildLog&_
> focus=379#_state=371
> [3] https://github.com/apache/ignite/blob/master/pom.xml#L533
> [4] https://ci.ignite.apache.org/viewLog.html?buildId=960221&bui
> ldTypeId=Ignite20Tests_IgniteRdd&tab=buildParameters
> [5] http://apache-ignite-developers.2346864.n4.nabble.com/
> Integration-of-Spark-and-Ignite-Prototype-tp22649p23099.html
> [6] https://ci.ignite.apache.org/viewLog.html?buildId=960220&bui
> ldTypeId=Ignite20Tests_IgniteRddSpark210&tab=testsInfo
> [7] https://github.com/apache/ignite/blob/master/modules/spark/
> src/test/scala/org/apache/ignite/spark/IgniteRDDSpec.scala
>