You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "Evans Ye (JIRA)" <ji...@apache.org> on 2019/03/18 14:54:00 UTC
[jira] [Comment Edited] (BIGTOP-3192) Remove Spark 1.6 from the
stack
[ https://issues.apache.org/jira/browse/BIGTOP-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795066#comment-16795066 ]
Evans Ye edited comment on BIGTOP-3192 at 3/18/19 2:53 PM:
-----------------------------------------------------------
Build for mahout and Phoenix are good. However crunch failed...
{code}
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/cr
unch/fn/SPairFlatMapFunction.java:[35,34] for-each not applicable to expression type
required: array or java.lang.Iterable
found: java.util.Iterator<scala.Tuple2<K,V>>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CrunchPairTuple2.java:[28,8] org.apache.crunch.impl.spark.fn.CrunchPairTuple2 is not abstract and does not override abstract method call(java.util.Iterator<org.apache.crunch.Pair<K,V>>) in org.apache.spark.api.java.function.PairFlatMapFunction
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CrunchPairTuple2.java:[30,33] call(java.util.Iterator<org.apache.crunch.Pair<K,V>>) in org.apache.crunch.impl.spark.fn.CrunchPairTuple2 cannot implement call(T) in org.apache.spark.api.java.function.PairFlatMapFunction
return type java.lang.Iterable<scala.Tuple2<K,V>> is not compatible with java.util.Iterator<scala.Tuple2<K,V>>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CrunchPairTuple2.java:[29,3] method does not override or implement a method from a supertype
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFlatMapFunction.java:[33,22] for-each not applicable to expression type
required: array or java.lang.Iterable
found: java.util.Iterator<R>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SDoubleFlatMapFunction.java:[33,27] for-each not applicable to expression type
required: array or java.lang.Iterable
found: java.util.Iterator<java.lang.Double>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/FlatMapPairDoFn.java:[30,8] org.apache.crunch.impl.spark.fn.FlatMapPairDoFn is not abstract and does not override abstract method call(java.util.Iterator<scala.Tuple2<K,V>>) in org.apache.spark.api.java.function.FlatMapFunction
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/FlatMapPairDoFn.java:[40,22] call(java.util.Iterator<scala.Tuple2<K,V>>) in org.apache.crunch.impl.spark.fn.FlatMapPairDoFn cannot implement call(T) in org.apache.spark.api.java.function.FlatMapFunction
return type java.lang.Iterable<T> is not compatible with java.util.Iterator<T>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/FlatMapPairDoFn.java:[39,3] method does not override or implement a method from a supertype
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CombineMapsideFunction.java:[35,8] org.apache.crunch.impl.spark.fn.CombineMapsideFunction is not abstract and does not override abstract method call(java.util.Iterator<scala.Tuple2<K,V>>) in org.apache.spark.api.java.function.PairFlatMapFunction
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CombineMapsideFunction.java:[48,33] call(java.util.Iterator<scala.Tuple2<K,V>>) in org.apache.crunch.impl.spark.fn.CombineMapsideFunction cannot implement call(T) in org.apache.spark.api.java.function.PairFlatMapFunction
return type java.lang.Iterable<scala.Tuple2<K,V>> is not compatible with java.util.Iterator<scala.Tuple2<K,V>>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CombineMapsideFunction.java:[47,3] method does not override or implement a method from a supertype
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFunctions.java:[63,41] <anonymous org.apache.crunch.fn.SFunctions$4> is not abstract and does not override abstract method call(T) in org.apache.spark.api.java.function.FlatMapFunction
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFunctions.java:[65,26] call(T) in <anonymous org.apache.crunch.fn.SFunctions$4> cannot implement call(T) in org.apache.spark.api.java.function.FlatMapFunction
return type java.lang.Iterable<R> is not compatible with java.util.Iterator<R>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFunctions.java:[64,7] method does not override or implement a method from a supertype
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFunctions.java:[66,22] incompatible types: java.util.Iterator<R> cannot be converted to java.lang.Iterable<R>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFlatMapFunction2.java:[35,22] for-each not applicable to expression type
required: array or java.lang.Iterable
found: java.util.Iterator<R>
{code}
Build for mahout and Phoenix are good. However crunch failed...
{code}
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/cr
unch/fn/SPairFlatMapFunction.java:[35,34] for-each not applicable to expression type
required: array or java.lang.Iterable
found: java.util.Iterator<scala.Tuple2<K,V>>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CrunchPairTuple2.java:[28,8] org.apache.crunch.impl.spark.fn.CrunchPairTuple2 is not abstract and does not override abstract method call(java.util.Iterator<org.apache.crunch.Pair<K,V>>) in org.apache.spark.api.java.function.PairFlatMapFunction
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CrunchPairTuple2.java:[30,33] call(java.util.Iterator<org.apache.crunch.Pair<K,V>>) in org.apache.crunch.impl.spark.fn.CrunchPairTuple2 cannot implement call(T) in org.apache.spark.api.java.function.PairFlatMapFunction
return type java.lang.Iterable<scala.Tuple2<K,V>> is not compatible with java.util.Iterator<scala.Tuple2<K,V>>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CrunchPairTuple2.java:[29,3] method does not override or implement a method from a supertype
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFlatMapFunction.java:[33,22] for-each not applicable to expression type
required: array or java.lang.Iterable
found: java.util.Iterator<R>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SDoubleFlatMapFunction.java:[33,27] for-each not applicable to expression type
required: array or java.lang.Iterable
found: java.util.Iterator<java.lang.Double>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/FlatMapPairDoFn.java:[30,8] org.apache.crunch.impl.spark.fn.FlatMapPairDoFn is not abstract and does not override abstract method call(java.util.Iterator<scala.Tuple2<K,V>>) in org.apache.spark.api.java.function.FlatMapFunction
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/FlatMapPairDoFn.java:[40,22] call(java.util.Iterator<scala.Tuple2<K,V>>) in org.apache.crunch.impl.spark.fn.FlatMapPairDoFn cannot implement call(T) in org.apache.spark.api.java.function.FlatMapFunction
return type java.lang.Iterable<T> is not compatible with java.util.Iterator<T>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/FlatMapPairDoFn.java:[39,3] method does not override or implement a method from a supertype
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CombineMapsideFunction.java:[35,8] org.apache.crunch.impl.spark.fn.CombineMapsideFunction is not abstract and does not override abstract method call(java.util.Iterator<scala.Tuple2<K,V>>) in org.apache.spark.api.java.function.PairFlatMapFunction
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CombineMapsideFunction.java:[48,33] call(java.util.Iterator<scala.Tuple2<K,V>>) in org.apache.crunch.impl.spark.fn.CombineMapsideFunction cannot implement call(T) in org.apache.spark.api.java.function.PairFlatMapFunction
return type java.lang.Iterable<scala.Tuple2<K,V>> is not compatible with java.util.Iterator<scala.Tuple2<K,V>>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CombineMapsideFunction.java:[47,3] method does not override or implement a method from a supertype
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFunctions.java:[63,41] <anonymous org.apache.crunch.fn.SFunctions$4> is not abstract and does not override abstract method call(T) in org.apache.spark.api.java.function.FlatMapFunction
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFunctions.java:[65,26] call(T) in <anonymous org.apache.crunch.fn.SFunctions$4> cannot implement call(T) in org.apache.spark.api.java.function.FlatMapFunction
return type java.lang.Iterable<R> is not compatible with java.util.Iterator<R>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFunctions.java:[64,7] method does not override or implement a method from a supertype
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFunctions.java:[66,22] incompatible types: java.util.Iterator<R> cannot be converted to java.lang.Iterable<R>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFlatMapFunction2.java:[35,22] for-each not applicable to expression type
required: array or java.lang.Iterable
found: java.util.Iterator<R>
{code}
Options:
1. Preserve Spark1 for Crunch => harmless
2. Remove Spark1 as well as Crunch => harmful
Do we have option 3? I prefer 1.
was (Author: evans_ye):
Build for mahout and Phoenix are good. However crunch failed...
{code}
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/cr
unch/fn/SPairFlatMapFunction.java:[35,34] for-each not applicable to expression type
required: array or java.lang.Iterable
found: java.util.Iterator<scala.Tuple2<K,V>>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CrunchPairTuple2.java:[28,8] org.apache.crunch.impl.spark.fn.CrunchPairTuple2 is not abstract and does not override abstract method call(java.util.Iterator<org.apache.crunch.Pair<K,V>>) in org.apache.spark.api.java.function.PairFlatMapFunction
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CrunchPairTuple2.java:[30,33] call(java.util.Iterator<org.apache.crunch.Pair<K,V>>) in org.apache.crunch.impl.spark.fn.CrunchPairTuple2 cannot implement call(T) in org.apache.spark.api.java.function.PairFlatMapFunction
return type java.lang.Iterable<scala.Tuple2<K,V>> is not compatible with java.util.Iterator<scala.Tuple2<K,V>>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CrunchPairTuple2.java:[29,3] method does not override or implement a method from a supertype
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFlatMapFunction.java:[33,22] for-each not applicable to expression type
required: array or java.lang.Iterable
found: java.util.Iterator<R>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SDoubleFlatMapFunction.java:[33,27] for-each not applicable to expression type
required: array or java.lang.Iterable
found: java.util.Iterator<java.lang.Double>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/FlatMapPairDoFn.java:[30,8] org.apache.crunch.impl.spark.fn.FlatMapPairDoFn is not abstract and does not override abstract method call(java.util.Iterator<scala.Tuple2<K,V>>) in org.apache.spark.api.java.function.FlatMapFunction
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/FlatMapPairDoFn.java:[40,22] call(java.util.Iterator<scala.Tuple2<K,V>>) in org.apache.crunch.impl.spark.fn.FlatMapPairDoFn cannot implement call(T) in org.apache.spark.api.java.function.FlatMapFunction
return type java.lang.Iterable<T> is not compatible with java.util.Iterator<T>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/FlatMapPairDoFn.java:[39,3] method does not override or implement a method from a supertype
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CombineMapsideFunction.java:[35,8] org.apache.crunch.impl.spark.fn.CombineMapsideFunction is not abstract and does not override abstract method call(java.util.Iterator<scala.Tuple2<K,V>>) in org.apache.spark.api.java.function.PairFlatMapFunction
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CombineMapsideFunction.java:[48,33] call(java.util.Iterator<scala.Tuple2<K,V>>) in org.apache.crunch.impl.spark.fn.CombineMapsideFunction cannot implement call(T) in org.apache.spark.api.java.function.PairFlatMapFunction
return type java.lang.Iterable<scala.Tuple2<K,V>> is not compatible with java.util.Iterator<scala.Tuple2<K,V>>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/impl/spark/fn/CombineMapsideFunction.java:[47,3] method does not override or implement a method from a supertype
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFunctions.java:[63,41] <anonymous org.apache.crunch.fn.SFunctions$4> is not abstract and does not override abstract method call(T) in org.apache.spark.api.java.function.FlatMapFunction
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFunctions.java:[65,26] call(T) in <anonymous org.apache.crunch.fn.SFunctions$4> cannot implement call(T) in org.apache.spark.api.java.function.FlatMapFunction
return type java.lang.Iterable<R> is not compatible with java.util.Iterator<R>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFunctions.java:[64,7] method does not override or implement a method from a supertype
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFunctions.java:[66,22] incompatible types: java.util.Iterator<R> cannot be converted to java.lang.Iterable<R>
[ERROR] /bigtop/build/crunch/rpm/BUILD/apache-crunch-0.15.0-src/crunch-spark/src/main/java/org/apache/crunch/fn/SFlatMapFunction2.java:[35,22] for-each not applicable to expression type
required: array or java.lang.Iterable
found: java.util.Iterator<R>
{code}
> Remove Spark 1.6 from the stack
> -------------------------------
>
> Key: BIGTOP-3192
> URL: https://issues.apache.org/jira/browse/BIGTOP-3192
> Project: Bigtop
> Issue Type: Sub-task
> Components: spark
> Affects Versions: 1.3.0
> Reporter: Konstantin Boudnik
> Assignee: Konstantin Boudnik
> Priority: Major
> Fix For: 1.4.0
>
> Attachments: 0001-BIGTOP-3192.-Remove-Spark-1.6-from-the-stack.patch
>
>
> Let's get rid of the outdated version of Spark. There's not much need to support it going forward.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)