You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Denis Magda <dm...@apache.org> on 2016/12/11 06:54:45 UTC
Re: Support for Spark 2.0
Upgraded Ignite Spark module to the latest Spark version. The transition was trivial.
However, I couldn’t fully execute Spark’s the test suite on TeamCity due to some maven dependency related issue. The issue never happens on my local machine.
Anton V., please have a look at it and suggest how to get rid of it. The details are in the ticket:
https://issues.apache.org/jira/browse/IGNITE-3710 <https://issues.apache.org/jira/browse/IGNITE-3710>
—
Denis
> On Sep 20, 2016, at 6:46 PM, Roman Shtykh <rs...@yahoo.com.INVALID> wrote:
>
> There is a bunch of requests from users for this upgrade.
> https://issues.apache.org/jira/browse/IGNITE-3822 and related.
>
> -Roman
>
>
> On Wednesday, September 21, 2016 5:14 AM, Denis Magda <dm...@gridgain.com> wrote:
>
>
> Is this the only issue we have with Spark 2.0?
> https://issues.apache.org/jira/browse/IGNITE-3596 <https://issues.apache.org/jira/browse/IGNITE-3596>
>
> —
> Denis
>
>> On Sep 20, 2016, at 11:54 AM, Vladimir Ozerov <vo...@gridgain.com> wrote:
>>
>> I think we can expect Spark 2.0 support to be added in one of the nearest
>> releases (not sure if it will be 1.8 or 2.0).
>>
>> On Mon, Sep 19, 2016 at 10:20 PM, Richard Siebeling <rs...@gmail.com>
>> wrote:
>>
>>> I'd wish someone could fix this, unfortunately I can't do anything about it
>>> in the near future...
>>>
>>> On Mon, Sep 19, 2016 at 3:20 PM, Phadnis, Varun <ph...@sky.optymyze.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> Just to be clear, I am not explicitly building the Hadoop edition. I get
>>>> the described error when I simply specify the Spark version:
>>>>
>>>> mvn clean package -Dspark.version=2.0.0 -DskipTests
>>>>
>>>> Thanks for the quick response!
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Sergey Kozlov [mailto:skozlov@gridgain.com]
>>>> Sent: 19 September 2016 05:56
>>>> To: dev@ignite.apache.org
>>>> Subject: Re: Support for Spark 2.0
>>>>
>>>> Hi
>>>>
>>>> It's a known issue IGNITE-3596 Hadoop edition can't be compiled against
>>>> spark 2.0.0 <https://issues.apache.org/jira/browse/IGNITE-3596>
>>>>
>>>> Unfortunately there's no progress yet
>>>>
>>>> On Mon, Sep 19, 2016 at 1:36 PM, Phadnis, Varun <
>>> phadnis@sky.optymyze.com>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Can someone please tell me when is the support for Spark 2.0 planned?
>>>>>
>>>>> Currently Ignite cannot be built for Spark 2.0. Attempting this yields
>>>>> the following error :
>>>>>
>>>>>
>>>>> ----------------------------------------------------------------------
>>>>> --
>>>>>
>>>>> [INFO] Building ignite-spark 1.8.0-SNAPSHOT [INFO]
>>>>>
>>>>> ----------------------------------------------------------------------
>>>>> --
>>>>>
>>>>> [INFO]
>>>>>
>>>>> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ ignite-spark
>>>>> --- [INFO] Deleting /home/spark/code/ignite/modules/spark/target
>>>>>
>>>>> [INFO]
>>>>>
>>>>> [INFO] --- flatten-maven-plugin:1.0.0-beta-3:clean
>>>>> (flatten.clean.before) @ ignite-spark --- [INFO] Deleting
>>>>> /home/spark/code/ignite/ modules/spark/pom-installed.xml
>>>>>
>>>>> [INFO]
>>>>>
>>>>> [INFO] --- maven-enforcer-plugin:1.4:enforce (default) @ ignite-spark
>>>>> --- [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process
>>>>> (default) @ ignite-spark --- [INFO] [INFO] ---
>>>>> maven-remote-resources-plugin:1.5:process
>>>>> (ignite-dependencies) @ ignite-spark --- [INFO] [INFO] ---
>>>>> maven-resources-plugin:2.6:resources (default-resources) @
>>>>> ignite-spark
>>>>> --- [INFO] Using 'UTF-8' encoding to copy filtered resources.
>>>>>
>>>>> [INFO] skip non existing resourceDirectory /home/spark/code/ignite/
>>>>> modules/spark/src/main/resources
>>>>>
>>>>> [INFO] Copying 4 resources
>>>>>
>>>>> [INFO] Copying 4 resources
>>>>>
>>>>> [INFO]
>>>>>
>>>>> [INFO] --- flatten-maven-plugin:1.0.0-beta-3:flatten (flatten) @
>>>>> ignite-spark --- [INFO] Generating flattened POM of project
>>>>> org.apache.ignite:ignite-spark:jar:1.8.0-SNAPSHOT...
>>>>>
>>>>> [WARNING] Ignoring multiple XML header comment!
>>>>>
>>>>> [INFO]
>>>>>
>>>>> [INFO] --- scala-maven-plugin:3.2.0:add-source (scala-compile-first) @
>>>>> ignite-spark --- [INFO] Add Source directory:
>>>>>
>>>>> /home/spark/code/ignite/modules/spark/src/main/scala
>>>>>
>>>>> [INFO] Add Test Source directory:
>>>>>
>>>>> /home/spark/code/ignite/modules/spark/src/test/scala
>>>>>
>>>>> [INFO]
>>>>>
>>>>> [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @
>>>>> ignite-spark --- [WARNING] Expected all dependencies to require Scala
>>>>> version: 2.11.7 [WARNING]
>>>>> org.apache.ignite:ignite-spark:1.8.0-SNAPSHOT
>>>>> requires scala
>>>>>
>>>>> version: 2.11.7
>>>>>
>>>>> [WARNING] com.twitter:chill_2.11:0.8.0 requires scala version: 2.11.7
>>>>> [WARNING] org.apache.spark:spark-core_2.11:2.0.0 requires scala
>>>> version:
>>>>>
>>>>> 2.11.8
>>>>>
>>>>> [WARNING] Multiple versions of scala libraries detected!
>>>>>
>>>>> [INFO] /home/spark/code/ignite/modules/spark/src/main/scala:-1: info:
>>>>>
>>>>> compiling
>>>>>
>>>>> [INFO] Compiling 8 source files to
>>>>>
>>>>> /home/spark/code/ignite/modules/spark/target/classes at 1474028222935
>>>>> [ERROR]
>>>>>
>>>>> /home/spark/code/ignite/modules/spark/src/main/scala/
>>>>> org/apache/ignite/spark/IgniteContext.scala:25:
>>>>>
>>>>> error: object Logging is not a member of package org.apache.spark
>>>>> [ERROR] import org.apache.spark.{Logging, SparkContext}
>>>>>
>>>>> [ERROR] ^
>>>>>
>>>>> [ERROR]
>>>>>
>>>>> /home/spark/code/ignite/modules/spark/src/main/scala/
>>>>> org/apache/ignite/spark/IgniteContext.scala:37:
>>>>>
>>>>> error: not found: type Logging
>>>>>
>>>>> [ERROR] ) extends Serializable with Logging {
>>>>>
>>>>> [ERROR] ^
>>>>>
>>>>> [ERROR]
>>>>>
>>>>> /home/spark/code/ignite/modules/spark/src/main/scala/
>>>>> org/apache/ignite/spark/IgniteContext.scala:50:
>>>>>
>>>>> error: not found: value logInfo
>>>>>
>>>>> [ERROR] logInfo("Will start Ignite nodes on " + workers + "
>>>>>
>>>>> workers")
>>>>>
>>>>> [ERROR] ^
>>>>>
>>>>> [ERROR]
>>>>>
>>>>> /home/spark/code/ignite/modules/spark/src/main/scala/
>>>>> org/apache/ignite/spark/IgniteContext.scala:129:
>>>>>
>>>>> error: not found: value logInfo
>>>>>
>>>>> [ERROR] logInfo("Setting IGNITE_HOME from driver not as it
>>> is
>>>>>
>>>>> not available on this worker: " + igniteHome)
>>>>>
>>>>> [ERROR] ^
>>>>>
>>>>> [ERROR]
>>>>>
>>>>> /home/spark/code/ignite/modules/spark/src/main/scala/
>>>>> org/apache/ignite/spark/IgniteContext.scala:146:
>>>>>
>>>>> error: not found: value logError
>>>>>
>>>>> [ERROR] logError("Failed to start Ignite.", e)
>>>>>
>>>>> [INFO] ^
>>>>>
>>>>> [ERROR]
>>>>>
>>>>> /home/spark/code/ignite/modules/spark/src/main/scala/
>>>>> org/apache/ignite/spark/IgniteContext.scala:164:
>>>>>
>>>>> error: not found: value logInfo
>>>>>
>>>>> [ERROR] logInfo("Will stop Ignite nodes on " + workers
>>> +
>>>> "
>>>>>
>>>>> workers")
>>>>>
>>>>> [ERROR] ^
>>>>>
>>>>> [ERROR] 6 errors found
>>>>>
>>>>> [INFO]
>>>>>
>>>>> ----------------------------------------------------------------------
>>>>> --
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sergey Kozlov
>>>> GridGain Systems
>>>> www.gridgain.com
>>>>
>>>
>
>