You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by "Denis Magda (JIRA)" <ji...@apache.org> on 2017/08/23 00:26:00 UTC

[jira] [Created] (IGNITE-6163) Upgrade to Spark 2.2.0

Denis Magda created IGNITE-6163:
-----------------------------------

             Summary: Upgrade to Spark 2.2.0
                 Key: IGNITE-6163
                 URL: https://issues.apache.org/jira/browse/IGNITE-6163
             Project: Ignite
          Issue Type: Improvement
            Reporter: Denis Magda
            Priority: Blocker
             Fix For: 2.2


Apache Ignite's Spark integration module has to be updated to Spark 2.2.0 to avoid issues like this:
https://github.com/dmagda/IgniteSparkIoT/issues/1

In short, Spark 2.1.0 and Spark 2.2.0 modules can't be mixed together, otherwise, the exceptions like this will pop up:

{code}
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.ui.SparkUI.setStreamingJobProgressListener(Lorg/apache/spark/scheduler/SparkListener;)V
	at org.apache.spark.streaming.ui.StreamingTab.<init>(StreamingTab.scala:41)
	at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:192)
	at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:85)
	at org.apache.spark.streaming.api.java.JavaStreamingContext.<init>(JavaStreamingContext.scala:138)
	at org.apache.ignite.iot.SparkStreamerStartup.main(SparkStreamerStartup.java:71)
17/08/22 18:27:25 INFO SparkContext: Invoking stop() from shutdown hook
17/08/22 18:27:25 INFO SparkUI: Stopped Spark web UI at http://10.0.1.6:4040
{code} 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Upgrade to Spark 2.2.0

Posted by Denis Magda <dm...@apache.org>.
Opps,

It’s time for Ignite to say goodbye to Java 7 too. Hope this happens in the beginning of the next year.

—
Denis

> On Nov 29, 2017, at 3:08 PM, Valentin Kulichenko <va...@gmail.com> wrote:
> 
> Denis,
> 
> I just found out that with version 2.2.0 Spark dropped Java 7 support:
> https://spark.apache.org/docs/2.2.0/
> 
> I'm not sure it's a good idea to do an upgrade as long as we support Java
> 7. Probably we should find a way to support both 2.1.0 and 2.2.0 in
> parallel. What do you think?
> 
> -Val
> 
> On Tue, Aug 22, 2017 at 5:30 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
> 
>> +1
>> 
>> On Tue, Aug 22, 2017 at 5:30 PM, Denis Magda <dm...@apache.org> wrote:
>> 
>>> Seems it’s time to upgrade ignite-spark module once again. Soon we should
>>> start getting more issues related to Spark 2.2.0 incompatibility.
>>> 
>>> —
>>> Denis
>>> 
>>>> Begin forwarded message:
>>>> 
>>>> From: "Denis Magda (JIRA)" <ji...@apache.org>
>>>> Subject: [jira] [Created] (IGNITE-6163) Upgrade to Spark 2.2.0
>>>> Date: August 22, 2017 at 5:26:00 PM PDT
>>>> To: dev@ignite.apache.org
>>>> Reply-To: dev@ignite.apache.org
>>>> 
>>>> https://issues.apache.org/jira/browse/IGNITE-6163 <
>>> https://issues.apache.org/jira/browse/IGNITE-6163>
>>> 
>>

Re: Upgrade to Spark 2.2.0

Posted by Valentin Kulichenko <va...@gmail.com>.
Denis,

I just found out that with version 2.2.0 Spark dropped Java 7 support:
https://spark.apache.org/docs/2.2.0/

I'm not sure it's a good idea to do an upgrade as long as we support Java
7. Probably we should find a way to support both 2.1.0 and 2.2.0 in
parallel. What do you think?

-Val

On Tue, Aug 22, 2017 at 5:30 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:

> +1
>
> On Tue, Aug 22, 2017 at 5:30 PM, Denis Magda <dm...@apache.org> wrote:
>
> > Seems it’s time to upgrade ignite-spark module once again. Soon we should
> > start getting more issues related to Spark 2.2.0 incompatibility.
> >
> > —
> > Denis
> >
> > > Begin forwarded message:
> > >
> > > From: "Denis Magda (JIRA)" <ji...@apache.org>
> > > Subject: [jira] [Created] (IGNITE-6163) Upgrade to Spark 2.2.0
> > > Date: August 22, 2017 at 5:26:00 PM PDT
> > > To: dev@ignite.apache.org
> > > Reply-To: dev@ignite.apache.org
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-6163 <
> > https://issues.apache.org/jira/browse/IGNITE-6163>
> >
>

Re: Upgrade to Spark 2.2.0

Posted by Dmitriy Setrakyan <ds...@apache.org>.
+1

On Tue, Aug 22, 2017 at 5:30 PM, Denis Magda <dm...@apache.org> wrote:

> Seems it’s time to upgrade ignite-spark module once again. Soon we should
> start getting more issues related to Spark 2.2.0 incompatibility.
>
> —
> Denis
>
> > Begin forwarded message:
> >
> > From: "Denis Magda (JIRA)" <ji...@apache.org>
> > Subject: [jira] [Created] (IGNITE-6163) Upgrade to Spark 2.2.0
> > Date: August 22, 2017 at 5:26:00 PM PDT
> > To: dev@ignite.apache.org
> > Reply-To: dev@ignite.apache.org
> >
> > https://issues.apache.org/jira/browse/IGNITE-6163 <
> https://issues.apache.org/jira/browse/IGNITE-6163>
>

Upgrade to Spark 2.2.0

Posted by Denis Magda <dm...@apache.org>.
Seems it’s time to upgrade ignite-spark module once again. Soon we should start getting more issues related to Spark 2.2.0 incompatibility.

—
Denis

> Begin forwarded message:
> 
> From: "Denis Magda (JIRA)" <ji...@apache.org>
> Subject: [jira] [Created] (IGNITE-6163) Upgrade to Spark 2.2.0
> Date: August 22, 2017 at 5:26:00 PM PDT
> To: dev@ignite.apache.org
> Reply-To: dev@ignite.apache.org
> 
> https://issues.apache.org/jira/browse/IGNITE-6163 <https://issues.apache.org/jira/browse/IGNITE-6163>