You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Alexey Zinoviev <za...@gmail.com> on 2019/10/14 11:00:26 UTC
Ignite-Spark integration meeting
Hi, Igniters, and especially Nikolay Izhikov
I suggest to arrange the meeting via ASF-slack in #ignite-spark-integration
channel at 17 October, in 16 MSK Time.
The language of the meeting: Russian
*The topics to discuss:*
- Review of next tickets
1. [Spark][Bug] IgniteSpark integration forget to close the
IgniteContext and stops the client node in case if error during
PairFunction logic
<https://issues.apache.org/jira/browse/IGNITE-11724>
2. [Spark] Add initial support of Spark 2.4
<https://issues.apache.org/jira/browse/IGNITE-12247>
3. [Spark] Add joins to Spark Dataframe examples
<https://issues.apache.org/jira/browse/IGNITE-9732>
- Known issues of migration to 2.4 version
- Organization of multiple version support for Apache Spark
P.S. meeting notes will be shared with the community
P.P.S Nikolay, if you have any objections about the time or topics, please
let me know
Re: Ignite-Spark integration meeting
Posted by Nikolay Izhikov <ni...@apache.org>.
The only issue with the Spark integration tests - they should be runned under JDK8.
That's all.
I wrote about it recently [1]
[1] http://apache-ignite-developers.2346864.n4.nabble.com/Spark-examples-on-TC-td43591.html
В Ср, 23/10/2019 в 17:46 +0200, Alexey Zinoviev пишет:
> Good point, now they are not in the Best state, we have a few ideas how to
> fix, will discuss it later here in this thread.
>
> If you have any objection write it, but my position - creation special
> Spark Visa on TC, like for ML, maybe two for different Spark versions, and
> all commits to Spark will require this Visa.
>
>
> ср, 23 окт. 2019 г., 16:25 Ivan Pavlukhin <vo...@gmail.com>:
>
> > Actually, I meant running tests on TeamCity.
> >
> > ср, 23 окт. 2019 г. в 15:36, Alexey Zinoviev <za...@gmail.com>:
> > >
> > > What kind of testing do you mean? JUnit, integration via example or
> > > something else?
> > >
> > > ср, 23 окт. 2019 г., 14:16 Ivan Pavlukhin <vo...@gmail.com>:
> > >
> > > > Correction.
> > > >
> > > > How are you going to TEST integrations with different Spark versions?
> > > >
> > > > ср, 23 окт. 2019 г. в 15:16, Ivan Pavlukhin <vo...@gmail.com>:
> > > > >
> > > > > Alexey,
> > > > >
> > > > > Thank you for sharing the summary!
> > > > >
> > > > > Small question. How are you going to integrations with different
> >
> > Spark
> > > > versions?
> > > > >
> > > > > чт, 17 окт. 2019 г. в 22:30, Denis Magda <dm...@apache.org>:
> > > > > >
> > > > > > >
> > > > > > > 1. We are going to release Apache Ignite 2.8 with limited
> >
> > support of
> > > > > > > Spark 2.4.4 (known issues are listed here
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-12054)
> > > > > >
> > > > > >
> > > > > > What does limited support mean? Is it about regressions (something
> >
> > that
> > > > > > worked before but will fail) or capabilities unsupported even for
> >
> > the
> > > > > > presently released version of the integration?
> > > > > >
> > > > > > -
> > > > > > Denis
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev <
> > > >
> > > > zaleslaw.sin@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > The meeting notes and results of discussion are next:
> > > > > > >
> > > > > > > 1. We are going to release Apache Ignite 2.8 with limited
> >
> > support
> > > > of
> > > > > > > Spark 2.4.4 (known issues are listed here
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-12054)
> > > > > > > 2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4
> >
> > with
> > > > known
> > > > > > > fixed bugs
> > > > > > > 3. We will create the developer-profiles spark_2_3 and
> >
> > spark_2_4
> > > > > > > separately from current scala profile
> > > > > > > 4. We are ready to move the Ignite-Spark integration to the
> > > >
> > > > separate
> > > > > > > module after release 2.8 with support of most actual Spark
> > > >
> > > > versions and
> > > > > > > with independent release cycle
> > > > > > > 5. We will work together on the next release roadmap with
> >
> > Nikolay
> > > > and
> > > > > > > all interested igniters to improve the current solution (add
> >
> > new
> > > > > > > features
> > > > > > > as thin client support, working with caches via binary object
> >
> > and
> > > > so on)
> > > > > > > 6. Nikolay is ready to help with the review of tickets as an
> > > >
> > > > author of
> > > > > > > this component
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <
> >
> > zaleslaw.sin@gmail.com
> > > > > :
> > > > > > >
> > > > > > > > Yes, let's use the Russian for short discussion, the meeting
> >
> > notes
> > > > will
> > > > > > > be
> > > > > > > > presented in English
> > > > > > > >
> > > > > > > > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <
> >
> > nizhikov@apache.org
> > > > > :
> > > > > > > >
> > > > > > > > > Hello, Alexey.
> > > > > > > > >
> > > > > > > > > I'm ready to take part in the meeting.
> > > > > > > > > It will be in Russian, isn't it?
> > > > > > > > >
> > > > > > > > > В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> > > > > > > > > > Hi, Igniters, and especially Nikolay Izhikov
> > > > > > > > > >
> > > > > > > > > > I suggest to arrange the meeting via ASF-slack in
> > > > > > > > >
> > > > > > > > > #ignite-spark-integration
> > > > > > > > > > channel at 17 October, in 16 MSK Time.
> > > > > > > > > > The language of the meeting: Russian
> > > > > > > > > >
> > > > > > > > > > *The topics to discuss:*
> > > > > > > > > >
> > > > > > > > > > - Review of next tickets
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 1. [Spark][Bug] IgniteSpark integration forget to close
> >
> > the
> > > > > > > > > > IgniteContext and stops the client node in case if
> >
> > error
> > > > during
> > > > > > > > > > PairFunction logic
> > > > > > > > > > <https://issues.apache.org/jira/browse/IGNITE-11724>
> > > > > > > > > > 2. [Spark] Add initial support of Spark 2.4
> > > > > > > > > > <https://issues.apache.org/jira/browse/IGNITE-12247>
> > > > > > > > > > 3. [Spark] Add joins to Spark Dataframe examples
> > > > > > > > > > <https://issues.apache.org/jira/browse/IGNITE-9732>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > - Known issues of migration to 2.4 version
> > > > > > > > > > - Organization of multiple version support for Apache
> >
> > Spark
> > > > > > > > > >
> > > > > > > > > > P.S. meeting notes will be shared with the community
> > > > > > > > > > P.P.S Nikolay, if you have any objections about the time or
> > > >
> > > > topics,
> > > > > > > > > please
> > > > > > > > > > let me know
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Ivan Pavlukhin
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >
Re: Ignite-Spark integration meeting
Posted by Alexey Zinoviev <za...@gmail.com>.
Good point, now they are not in the Best state, we have a few ideas how to
fix, will discuss it later here in this thread.
If you have any objection write it, but my position - creation special
Spark Visa on TC, like for ML, maybe two for different Spark versions, and
all commits to Spark will require this Visa.
ср, 23 окт. 2019 г., 16:25 Ivan Pavlukhin <vo...@gmail.com>:
> Actually, I meant running tests on TeamCity.
>
> ср, 23 окт. 2019 г. в 15:36, Alexey Zinoviev <za...@gmail.com>:
> >
> > What kind of testing do you mean? JUnit, integration via example or
> > something else?
> >
> > ср, 23 окт. 2019 г., 14:16 Ivan Pavlukhin <vo...@gmail.com>:
> >
> > > Correction.
> > >
> > > How are you going to TEST integrations with different Spark versions?
> > >
> > > ср, 23 окт. 2019 г. в 15:16, Ivan Pavlukhin <vo...@gmail.com>:
> > > >
> > > > Alexey,
> > > >
> > > > Thank you for sharing the summary!
> > > >
> > > > Small question. How are you going to integrations with different
> Spark
> > > versions?
> > > >
> > > > чт, 17 окт. 2019 г. в 22:30, Denis Magda <dm...@apache.org>:
> > > > >
> > > > > >
> > > > > > 1. We are going to release Apache Ignite 2.8 with limited
> support of
> > > > > > Spark 2.4.4 (known issues are listed here
> > > > > > https://issues.apache.org/jira/browse/IGNITE-12054)
> > > > >
> > > > >
> > > > > What does limited support mean? Is it about regressions (something
> that
> > > > > worked before but will fail) or capabilities unsupported even for
> the
> > > > > presently released version of the integration?
> > > > >
> > > > > -
> > > > > Denis
> > > > >
> > > > >
> > > > > On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev <
> > > zaleslaw.sin@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > The meeting notes and results of discussion are next:
> > > > > >
> > > > > > 1. We are going to release Apache Ignite 2.8 with limited
> support
> > > of
> > > > > > Spark 2.4.4 (known issues are listed here
> > > > > > https://issues.apache.org/jira/browse/IGNITE-12054)
> > > > > > 2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4
> with
> > > known
> > > > > > fixed bugs
> > > > > > 3. We will create the developer-profiles spark_2_3 and
> spark_2_4
> > > > > > separately from current scala profile
> > > > > > 4. We are ready to move the Ignite-Spark integration to the
> > > separate
> > > > > > module after release 2.8 with support of most actual Spark
> > > versions and
> > > > > > with independent release cycle
> > > > > > 5. We will work together on the next release roadmap with
> Nikolay
> > > and
> > > > > > all interested igniters to improve the current solution (add
> new
> > > > > > features
> > > > > > as thin client support, working with caches via binary object
> and
> > > so on)
> > > > > > 6. Nikolay is ready to help with the review of tickets as an
> > > author of
> > > > > > this component
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > >
> > > > > >
> > > > > > пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <
> zaleslaw.sin@gmail.com
> > > >:
> > > > > >
> > > > > > > Yes, let's use the Russian for short discussion, the meeting
> notes
> > > will
> > > > > > be
> > > > > > > presented in English
> > > > > > >
> > > > > > > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <
> nizhikov@apache.org
> > > >:
> > > > > > >
> > > > > > >> Hello, Alexey.
> > > > > > >>
> > > > > > >> I'm ready to take part in the meeting.
> > > > > > >> It will be in Russian, isn't it?
> > > > > > >>
> > > > > > >> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> > > > > > >> > Hi, Igniters, and especially Nikolay Izhikov
> > > > > > >> >
> > > > > > >> > I suggest to arrange the meeting via ASF-slack in
> > > > > > >> #ignite-spark-integration
> > > > > > >> > channel at 17 October, in 16 MSK Time.
> > > > > > >> > The language of the meeting: Russian
> > > > > > >> >
> > > > > > >> > *The topics to discuss:*
> > > > > > >> >
> > > > > > >> > - Review of next tickets
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > 1. [Spark][Bug] IgniteSpark integration forget to close
> the
> > > > > > >> > IgniteContext and stops the client node in case if
> error
> > > during
> > > > > > >> > PairFunction logic
> > > > > > >> > <https://issues.apache.org/jira/browse/IGNITE-11724>
> > > > > > >> > 2. [Spark] Add initial support of Spark 2.4
> > > > > > >> > <https://issues.apache.org/jira/browse/IGNITE-12247>
> > > > > > >> > 3. [Spark] Add joins to Spark Dataframe examples
> > > > > > >> > <https://issues.apache.org/jira/browse/IGNITE-9732>
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > - Known issues of migration to 2.4 version
> > > > > > >> > - Organization of multiple version support for Apache
> Spark
> > > > > > >> >
> > > > > > >> > P.S. meeting notes will be shared with the community
> > > > > > >> > P.P.S Nikolay, if you have any objections about the time or
> > > topics,
> > > > > > >> please
> > > > > > >> > let me know
> > > > > > >>
> > > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>
Re: Ignite-Spark integration meeting
Posted by Ivan Pavlukhin <vo...@gmail.com>.
Actually, I meant running tests on TeamCity.
ср, 23 окт. 2019 г. в 15:36, Alexey Zinoviev <za...@gmail.com>:
>
> What kind of testing do you mean? JUnit, integration via example or
> something else?
>
> ср, 23 окт. 2019 г., 14:16 Ivan Pavlukhin <vo...@gmail.com>:
>
> > Correction.
> >
> > How are you going to TEST integrations with different Spark versions?
> >
> > ср, 23 окт. 2019 г. в 15:16, Ivan Pavlukhin <vo...@gmail.com>:
> > >
> > > Alexey,
> > >
> > > Thank you for sharing the summary!
> > >
> > > Small question. How are you going to integrations with different Spark
> > versions?
> > >
> > > чт, 17 окт. 2019 г. в 22:30, Denis Magda <dm...@apache.org>:
> > > >
> > > > >
> > > > > 1. We are going to release Apache Ignite 2.8 with limited support of
> > > > > Spark 2.4.4 (known issues are listed here
> > > > > https://issues.apache.org/jira/browse/IGNITE-12054)
> > > >
> > > >
> > > > What does limited support mean? Is it about regressions (something that
> > > > worked before but will fail) or capabilities unsupported even for the
> > > > presently released version of the integration?
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev <
> > zaleslaw.sin@gmail.com>
> > > > wrote:
> > > >
> > > > > The meeting notes and results of discussion are next:
> > > > >
> > > > > 1. We are going to release Apache Ignite 2.8 with limited support
> > of
> > > > > Spark 2.4.4 (known issues are listed here
> > > > > https://issues.apache.org/jira/browse/IGNITE-12054)
> > > > > 2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4 with
> > known
> > > > > fixed bugs
> > > > > 3. We will create the developer-profiles spark_2_3 and spark_2_4
> > > > > separately from current scala profile
> > > > > 4. We are ready to move the Ignite-Spark integration to the
> > separate
> > > > > module after release 2.8 with support of most actual Spark
> > versions and
> > > > > with independent release cycle
> > > > > 5. We will work together on the next release roadmap with Nikolay
> > and
> > > > > all interested igniters to improve the current solution (add new
> > > > > features
> > > > > as thin client support, working with caches via binary object and
> > so on)
> > > > > 6. Nikolay is ready to help with the review of tickets as an
> > author of
> > > > > this component
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > > >
> > > > > пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <zaleslaw.sin@gmail.com
> > >:
> > > > >
> > > > > > Yes, let's use the Russian for short discussion, the meeting notes
> > will
> > > > > be
> > > > > > presented in English
> > > > > >
> > > > > > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <nizhikov@apache.org
> > >:
> > > > > >
> > > > > >> Hello, Alexey.
> > > > > >>
> > > > > >> I'm ready to take part in the meeting.
> > > > > >> It will be in Russian, isn't it?
> > > > > >>
> > > > > >> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> > > > > >> > Hi, Igniters, and especially Nikolay Izhikov
> > > > > >> >
> > > > > >> > I suggest to arrange the meeting via ASF-slack in
> > > > > >> #ignite-spark-integration
> > > > > >> > channel at 17 October, in 16 MSK Time.
> > > > > >> > The language of the meeting: Russian
> > > > > >> >
> > > > > >> > *The topics to discuss:*
> > > > > >> >
> > > > > >> > - Review of next tickets
> > > > > >> >
> > > > > >> >
> > > > > >> > 1. [Spark][Bug] IgniteSpark integration forget to close the
> > > > > >> > IgniteContext and stops the client node in case if error
> > during
> > > > > >> > PairFunction logic
> > > > > >> > <https://issues.apache.org/jira/browse/IGNITE-11724>
> > > > > >> > 2. [Spark] Add initial support of Spark 2.4
> > > > > >> > <https://issues.apache.org/jira/browse/IGNITE-12247>
> > > > > >> > 3. [Spark] Add joins to Spark Dataframe examples
> > > > > >> > <https://issues.apache.org/jira/browse/IGNITE-9732>
> > > > > >> >
> > > > > >> >
> > > > > >> > - Known issues of migration to 2.4 version
> > > > > >> > - Organization of multiple version support for Apache Spark
> > > > > >> >
> > > > > >> > P.S. meeting notes will be shared with the community
> > > > > >> > P.P.S Nikolay, if you have any objections about the time or
> > topics,
> > > > > >> please
> > > > > >> > let me know
> > > > > >>
> > > > > >
> > > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >
--
Best regards,
Ivan Pavlukhin
Re: Ignite-Spark integration meeting
Posted by Alexey Zinoviev <za...@gmail.com>.
What kind of testing do you mean? JUnit, integration via example or
something else?
ср, 23 окт. 2019 г., 14:16 Ivan Pavlukhin <vo...@gmail.com>:
> Correction.
>
> How are you going to TEST integrations with different Spark versions?
>
> ср, 23 окт. 2019 г. в 15:16, Ivan Pavlukhin <vo...@gmail.com>:
> >
> > Alexey,
> >
> > Thank you for sharing the summary!
> >
> > Small question. How are you going to integrations with different Spark
> versions?
> >
> > чт, 17 окт. 2019 г. в 22:30, Denis Magda <dm...@apache.org>:
> > >
> > > >
> > > > 1. We are going to release Apache Ignite 2.8 with limited support of
> > > > Spark 2.4.4 (known issues are listed here
> > > > https://issues.apache.org/jira/browse/IGNITE-12054)
> > >
> > >
> > > What does limited support mean? Is it about regressions (something that
> > > worked before but will fail) or capabilities unsupported even for the
> > > presently released version of the integration?
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev <
> zaleslaw.sin@gmail.com>
> > > wrote:
> > >
> > > > The meeting notes and results of discussion are next:
> > > >
> > > > 1. We are going to release Apache Ignite 2.8 with limited support
> of
> > > > Spark 2.4.4 (known issues are listed here
> > > > https://issues.apache.org/jira/browse/IGNITE-12054)
> > > > 2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4 with
> known
> > > > fixed bugs
> > > > 3. We will create the developer-profiles spark_2_3 and spark_2_4
> > > > separately from current scala profile
> > > > 4. We are ready to move the Ignite-Spark integration to the
> separate
> > > > module after release 2.8 with support of most actual Spark
> versions and
> > > > with independent release cycle
> > > > 5. We will work together on the next release roadmap with Nikolay
> and
> > > > all interested igniters to improve the current solution (add new
> > > > features
> > > > as thin client support, working with caches via binary object and
> so on)
> > > > 6. Nikolay is ready to help with the review of tickets as an
> author of
> > > > this component
> > > >
> > > > Thanks
> > > >
> > > >
> > > >
> > > > пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <zaleslaw.sin@gmail.com
> >:
> > > >
> > > > > Yes, let's use the Russian for short discussion, the meeting notes
> will
> > > > be
> > > > > presented in English
> > > > >
> > > > > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <nizhikov@apache.org
> >:
> > > > >
> > > > >> Hello, Alexey.
> > > > >>
> > > > >> I'm ready to take part in the meeting.
> > > > >> It will be in Russian, isn't it?
> > > > >>
> > > > >> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> > > > >> > Hi, Igniters, and especially Nikolay Izhikov
> > > > >> >
> > > > >> > I suggest to arrange the meeting via ASF-slack in
> > > > >> #ignite-spark-integration
> > > > >> > channel at 17 October, in 16 MSK Time.
> > > > >> > The language of the meeting: Russian
> > > > >> >
> > > > >> > *The topics to discuss:*
> > > > >> >
> > > > >> > - Review of next tickets
> > > > >> >
> > > > >> >
> > > > >> > 1. [Spark][Bug] IgniteSpark integration forget to close the
> > > > >> > IgniteContext and stops the client node in case if error
> during
> > > > >> > PairFunction logic
> > > > >> > <https://issues.apache.org/jira/browse/IGNITE-11724>
> > > > >> > 2. [Spark] Add initial support of Spark 2.4
> > > > >> > <https://issues.apache.org/jira/browse/IGNITE-12247>
> > > > >> > 3. [Spark] Add joins to Spark Dataframe examples
> > > > >> > <https://issues.apache.org/jira/browse/IGNITE-9732>
> > > > >> >
> > > > >> >
> > > > >> > - Known issues of migration to 2.4 version
> > > > >> > - Organization of multiple version support for Apache Spark
> > > > >> >
> > > > >> > P.S. meeting notes will be shared with the community
> > > > >> > P.P.S Nikolay, if you have any objections about the time or
> topics,
> > > > >> please
> > > > >> > let me know
> > > > >>
> > > > >
> > > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>
Re: Ignite-Spark integration meeting
Posted by Ivan Pavlukhin <vo...@gmail.com>.
Correction.
How are you going to TEST integrations with different Spark versions?
ср, 23 окт. 2019 г. в 15:16, Ivan Pavlukhin <vo...@gmail.com>:
>
> Alexey,
>
> Thank you for sharing the summary!
>
> Small question. How are you going to integrations with different Spark versions?
>
> чт, 17 окт. 2019 г. в 22:30, Denis Magda <dm...@apache.org>:
> >
> > >
> > > 1. We are going to release Apache Ignite 2.8 with limited support of
> > > Spark 2.4.4 (known issues are listed here
> > > https://issues.apache.org/jira/browse/IGNITE-12054)
> >
> >
> > What does limited support mean? Is it about regressions (something that
> > worked before but will fail) or capabilities unsupported even for the
> > presently released version of the integration?
> >
> > -
> > Denis
> >
> >
> > On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev <za...@gmail.com>
> > wrote:
> >
> > > The meeting notes and results of discussion are next:
> > >
> > > 1. We are going to release Apache Ignite 2.8 with limited support of
> > > Spark 2.4.4 (known issues are listed here
> > > https://issues.apache.org/jira/browse/IGNITE-12054)
> > > 2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4 with known
> > > fixed bugs
> > > 3. We will create the developer-profiles spark_2_3 and spark_2_4
> > > separately from current scala profile
> > > 4. We are ready to move the Ignite-Spark integration to the separate
> > > module after release 2.8 with support of most actual Spark versions and
> > > with independent release cycle
> > > 5. We will work together on the next release roadmap with Nikolay and
> > > all interested igniters to improve the current solution (add new
> > > features
> > > as thin client support, working with caches via binary object and so on)
> > > 6. Nikolay is ready to help with the review of tickets as an author of
> > > this component
> > >
> > > Thanks
> > >
> > >
> > >
> > > пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <za...@gmail.com>:
> > >
> > > > Yes, let's use the Russian for short discussion, the meeting notes will
> > > be
> > > > presented in English
> > > >
> > > > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <ni...@apache.org>:
> > > >
> > > >> Hello, Alexey.
> > > >>
> > > >> I'm ready to take part in the meeting.
> > > >> It will be in Russian, isn't it?
> > > >>
> > > >> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> > > >> > Hi, Igniters, and especially Nikolay Izhikov
> > > >> >
> > > >> > I suggest to arrange the meeting via ASF-slack in
> > > >> #ignite-spark-integration
> > > >> > channel at 17 October, in 16 MSK Time.
> > > >> > The language of the meeting: Russian
> > > >> >
> > > >> > *The topics to discuss:*
> > > >> >
> > > >> > - Review of next tickets
> > > >> >
> > > >> >
> > > >> > 1. [Spark][Bug] IgniteSpark integration forget to close the
> > > >> > IgniteContext and stops the client node in case if error during
> > > >> > PairFunction logic
> > > >> > <https://issues.apache.org/jira/browse/IGNITE-11724>
> > > >> > 2. [Spark] Add initial support of Spark 2.4
> > > >> > <https://issues.apache.org/jira/browse/IGNITE-12247>
> > > >> > 3. [Spark] Add joins to Spark Dataframe examples
> > > >> > <https://issues.apache.org/jira/browse/IGNITE-9732>
> > > >> >
> > > >> >
> > > >> > - Known issues of migration to 2.4 version
> > > >> > - Organization of multiple version support for Apache Spark
> > > >> >
> > > >> > P.S. meeting notes will be shared with the community
> > > >> > P.P.S Nikolay, if you have any objections about the time or topics,
> > > >> please
> > > >> > let me know
> > > >>
> > > >
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
--
Best regards,
Ivan Pavlukhin
Re: Ignite-Spark integration meeting
Posted by Ivan Pavlukhin <vo...@gmail.com>.
Alexey,
Thank you for sharing the summary!
Small question. How are you going to integrations with different Spark versions?
чт, 17 окт. 2019 г. в 22:30, Denis Magda <dm...@apache.org>:
>
> >
> > 1. We are going to release Apache Ignite 2.8 with limited support of
> > Spark 2.4.4 (known issues are listed here
> > https://issues.apache.org/jira/browse/IGNITE-12054)
>
>
> What does limited support mean? Is it about regressions (something that
> worked before but will fail) or capabilities unsupported even for the
> presently released version of the integration?
>
> -
> Denis
>
>
> On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev <za...@gmail.com>
> wrote:
>
> > The meeting notes and results of discussion are next:
> >
> > 1. We are going to release Apache Ignite 2.8 with limited support of
> > Spark 2.4.4 (known issues are listed here
> > https://issues.apache.org/jira/browse/IGNITE-12054)
> > 2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4 with known
> > fixed bugs
> > 3. We will create the developer-profiles spark_2_3 and spark_2_4
> > separately from current scala profile
> > 4. We are ready to move the Ignite-Spark integration to the separate
> > module after release 2.8 with support of most actual Spark versions and
> > with independent release cycle
> > 5. We will work together on the next release roadmap with Nikolay and
> > all interested igniters to improve the current solution (add new
> > features
> > as thin client support, working with caches via binary object and so on)
> > 6. Nikolay is ready to help with the review of tickets as an author of
> > this component
> >
> > Thanks
> >
> >
> >
> > пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <za...@gmail.com>:
> >
> > > Yes, let's use the Russian for short discussion, the meeting notes will
> > be
> > > presented in English
> > >
> > > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <ni...@apache.org>:
> > >
> > >> Hello, Alexey.
> > >>
> > >> I'm ready to take part in the meeting.
> > >> It will be in Russian, isn't it?
> > >>
> > >> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> > >> > Hi, Igniters, and especially Nikolay Izhikov
> > >> >
> > >> > I suggest to arrange the meeting via ASF-slack in
> > >> #ignite-spark-integration
> > >> > channel at 17 October, in 16 MSK Time.
> > >> > The language of the meeting: Russian
> > >> >
> > >> > *The topics to discuss:*
> > >> >
> > >> > - Review of next tickets
> > >> >
> > >> >
> > >> > 1. [Spark][Bug] IgniteSpark integration forget to close the
> > >> > IgniteContext and stops the client node in case if error during
> > >> > PairFunction logic
> > >> > <https://issues.apache.org/jira/browse/IGNITE-11724>
> > >> > 2. [Spark] Add initial support of Spark 2.4
> > >> > <https://issues.apache.org/jira/browse/IGNITE-12247>
> > >> > 3. [Spark] Add joins to Spark Dataframe examples
> > >> > <https://issues.apache.org/jira/browse/IGNITE-9732>
> > >> >
> > >> >
> > >> > - Known issues of migration to 2.4 version
> > >> > - Organization of multiple version support for Apache Spark
> > >> >
> > >> > P.S. meeting notes will be shared with the community
> > >> > P.P.S Nikolay, if you have any objections about the time or topics,
> > >> please
> > >> > let me know
> > >>
> > >
> >
--
Best regards,
Ivan Pavlukhin
Re: Ignite-Spark integration meeting
Posted by Denis Magda <dm...@apache.org>.
>
> 1. We are going to release Apache Ignite 2.8 with limited support of
> Spark 2.4.4 (known issues are listed here
> https://issues.apache.org/jira/browse/IGNITE-12054)
What does limited support mean? Is it about regressions (something that
worked before but will fail) or capabilities unsupported even for the
presently released version of the integration?
-
Denis
On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev <za...@gmail.com>
wrote:
> The meeting notes and results of discussion are next:
>
> 1. We are going to release Apache Ignite 2.8 with limited support of
> Spark 2.4.4 (known issues are listed here
> https://issues.apache.org/jira/browse/IGNITE-12054)
> 2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4 with known
> fixed bugs
> 3. We will create the developer-profiles spark_2_3 and spark_2_4
> separately from current scala profile
> 4. We are ready to move the Ignite-Spark integration to the separate
> module after release 2.8 with support of most actual Spark versions and
> with independent release cycle
> 5. We will work together on the next release roadmap with Nikolay and
> all interested igniters to improve the current solution (add new
> features
> as thin client support, working with caches via binary object and so on)
> 6. Nikolay is ready to help with the review of tickets as an author of
> this component
>
> Thanks
>
>
>
> пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <za...@gmail.com>:
>
> > Yes, let's use the Russian for short discussion, the meeting notes will
> be
> > presented in English
> >
> > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <ni...@apache.org>:
> >
> >> Hello, Alexey.
> >>
> >> I'm ready to take part in the meeting.
> >> It will be in Russian, isn't it?
> >>
> >> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> >> > Hi, Igniters, and especially Nikolay Izhikov
> >> >
> >> > I suggest to arrange the meeting via ASF-slack in
> >> #ignite-spark-integration
> >> > channel at 17 October, in 16 MSK Time.
> >> > The language of the meeting: Russian
> >> >
> >> > *The topics to discuss:*
> >> >
> >> > - Review of next tickets
> >> >
> >> >
> >> > 1. [Spark][Bug] IgniteSpark integration forget to close the
> >> > IgniteContext and stops the client node in case if error during
> >> > PairFunction logic
> >> > <https://issues.apache.org/jira/browse/IGNITE-11724>
> >> > 2. [Spark] Add initial support of Spark 2.4
> >> > <https://issues.apache.org/jira/browse/IGNITE-12247>
> >> > 3. [Spark] Add joins to Spark Dataframe examples
> >> > <https://issues.apache.org/jira/browse/IGNITE-9732>
> >> >
> >> >
> >> > - Known issues of migration to 2.4 version
> >> > - Organization of multiple version support for Apache Spark
> >> >
> >> > P.S. meeting notes will be shared with the community
> >> > P.P.S Nikolay, if you have any objections about the time or topics,
> >> please
> >> > let me know
> >>
> >
>
Re: Ignite-Spark integration meeting
Posted by Alexey Zinoviev <za...@gmail.com>.
The meeting notes and results of discussion are next:
1. We are going to release Apache Ignite 2.8 with limited support of
Spark 2.4.4 (known issues are listed here
https://issues.apache.org/jira/browse/IGNITE-12054)
2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4 with known
fixed bugs
3. We will create the developer-profiles spark_2_3 and spark_2_4
separately from current scala profile
4. We are ready to move the Ignite-Spark integration to the separate
module after release 2.8 with support of most actual Spark versions and
with independent release cycle
5. We will work together on the next release roadmap with Nikolay and
all interested igniters to improve the current solution (add new features
as thin client support, working with caches via binary object and so on)
6. Nikolay is ready to help with the review of tickets as an author of
this component
Thanks
пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev <za...@gmail.com>:
> Yes, let's use the Russian for short discussion, the meeting notes will be
> presented in English
>
> пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <ni...@apache.org>:
>
>> Hello, Alexey.
>>
>> I'm ready to take part in the meeting.
>> It will be in Russian, isn't it?
>>
>> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
>> > Hi, Igniters, and especially Nikolay Izhikov
>> >
>> > I suggest to arrange the meeting via ASF-slack in
>> #ignite-spark-integration
>> > channel at 17 October, in 16 MSK Time.
>> > The language of the meeting: Russian
>> >
>> > *The topics to discuss:*
>> >
>> > - Review of next tickets
>> >
>> >
>> > 1. [Spark][Bug] IgniteSpark integration forget to close the
>> > IgniteContext and stops the client node in case if error during
>> > PairFunction logic
>> > <https://issues.apache.org/jira/browse/IGNITE-11724>
>> > 2. [Spark] Add initial support of Spark 2.4
>> > <https://issues.apache.org/jira/browse/IGNITE-12247>
>> > 3. [Spark] Add joins to Spark Dataframe examples
>> > <https://issues.apache.org/jira/browse/IGNITE-9732>
>> >
>> >
>> > - Known issues of migration to 2.4 version
>> > - Organization of multiple version support for Apache Spark
>> >
>> > P.S. meeting notes will be shared with the community
>> > P.P.S Nikolay, if you have any objections about the time or topics,
>> please
>> > let me know
>>
>
Re: Ignite-Spark integration meeting
Posted by Alexey Zinoviev <za...@gmail.com>.
Yes, let's use the Russian for short discussion, the meeting notes will be
presented in English
пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov <ni...@apache.org>:
> Hello, Alexey.
>
> I'm ready to take part in the meeting.
> It will be in Russian, isn't it?
>
> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> > Hi, Igniters, and especially Nikolay Izhikov
> >
> > I suggest to arrange the meeting via ASF-slack in
> #ignite-spark-integration
> > channel at 17 October, in 16 MSK Time.
> > The language of the meeting: Russian
> >
> > *The topics to discuss:*
> >
> > - Review of next tickets
> >
> >
> > 1. [Spark][Bug] IgniteSpark integration forget to close the
> > IgniteContext and stops the client node in case if error during
> > PairFunction logic
> > <https://issues.apache.org/jira/browse/IGNITE-11724>
> > 2. [Spark] Add initial support of Spark 2.4
> > <https://issues.apache.org/jira/browse/IGNITE-12247>
> > 3. [Spark] Add joins to Spark Dataframe examples
> > <https://issues.apache.org/jira/browse/IGNITE-9732>
> >
> >
> > - Known issues of migration to 2.4 version
> > - Organization of multiple version support for Apache Spark
> >
> > P.S. meeting notes will be shared with the community
> > P.P.S Nikolay, if you have any objections about the time or topics,
> please
> > let me know
>
Re: Ignite-Spark integration meeting
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Alexey.
I'm ready to take part in the meeting.
It will be in Russian, isn't it?
В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> Hi, Igniters, and especially Nikolay Izhikov
>
> I suggest to arrange the meeting via ASF-slack in #ignite-spark-integration
> channel at 17 October, in 16 MSK Time.
> The language of the meeting: Russian
>
> *The topics to discuss:*
>
> - Review of next tickets
>
>
> 1. [Spark][Bug] IgniteSpark integration forget to close the
> IgniteContext and stops the client node in case if error during
> PairFunction logic
> <https://issues.apache.org/jira/browse/IGNITE-11724>
> 2. [Spark] Add initial support of Spark 2.4
> <https://issues.apache.org/jira/browse/IGNITE-12247>
> 3. [Spark] Add joins to Spark Dataframe examples
> <https://issues.apache.org/jira/browse/IGNITE-9732>
>
>
> - Known issues of migration to 2.4 version
> - Organization of multiple version support for Apache Spark
>
> P.S. meeting notes will be shared with the community
> P.P.S Nikolay, if you have any objections about the time or topics, please
> let me know