You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "dingwei2019 (JIRA)" <ji...@apache.org> on 2019/04/22 03:12:00 UTC
[jira] [Updated] (SPARK-27537)
spark-2.4.1/mlib-local/src/test/scala/org/apache/spark/ml/linalg/MatricesSuite.scala.866:Value
size is not a member of Object
[ https://issues.apache.org/jira/browse/SPARK-27537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dingwei2019 updated SPARK-27537:
--------------------------------
Docs Text: (was: the question is found in spark ml test module, althrough this is an test module, i want to figure it out.
from the describe above, it seems an incompatible problem between java 11 and scala 2.11.12.
if I change my jdk to jdk8, and there is no problem.
Below is my analysis:
it seems in spark if a method has implementation in java, spark will use java method, or will use scala method.
'string' class in java11 adds the lines method. This method conflicts with the scala syntax.
scala has lines method in 'stringlike' class, the method return an Iterator;
Iterator in scala has a toArray method, the method return an Array;
the class array in scala has a size method. so if spark use scala method, it will have no problem.
lines(Iterator)-->toArray(Array)-->size
But Java11 adds lines method in 'string', this will return a Stream;
Stream in java11 has toArray method, and will return Object;
Object has no 'size' method. This is what the error says.
(Stream)-->(Object)toArray-->has no size method.)
> spark-2.4.1/mlib-local/src/test/scala/org/apache/spark/ml/linalg/MatricesSuite.scala.866:Value size is not a member of Object
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-27537
> URL: https://issues.apache.org/jira/browse/SPARK-27537
> Project: Spark
> Issue Type: Bug
> Components: MLlib
> Affects Versions: 2.3.0, 2.4.1
> Environment: Machine：aarch64
> OS：Red Hat Enterprise Linux Server release 7.4
> Kernel：4.11.0-44.el7a
> spark version： spark-2.4.1
> java：openjdk version "11.0.2" 2019-01-15
> OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.2+9)
> scala：2.11.12
> gcc version：4.8.5
> Reporter: dingwei2019
> Priority: Major
> Labels: build, test
>
> [ERROR]: [Error] $SPARK_HOME/mlib-local/src/test/scala/org/apache/spark/ml/linalg/MatricesSuite.scala.866:Value size is not a member of Object
> [ERROR]: [Error] $SPARK_HOME/mlib-local/src/test/scala/org/apache/spark/ml/linalg/MatricesSuite.scala.869:Value size is not a member of Object
> ERROR: two errors found
> Below is the related code:
> 856 test("toString") {
> 857 val empty = Matrices.ones(0, 0)
> 858 empty.toString(0, 0)
> 859
> 860 val mat = Matrices.rand(5, 10, new Random())
> 861 mat.toString(-1, -5)
> 862 mat.toString(0, 0)
> 863 mat.toString(Int.MinValue, Int.MinValue)
> 864 mat.toString(Int.MaxValue, Int.MaxValue)
> 865 var lines = mat.toString(6, 50).lines.toArray
> 866 assert(lines.size == 5 && lines.forall(_.size <= 50))
> 867
> 868 lines = mat.toString(5, 100).lines.toArray
> 869 assert(lines.size == 5 && lines.forall(_.size <= 100))
> 870 }
> 871
> 872 test("numNonzeros and numActives") {
> 873 val dm1 = Matrices.dense(3, 2, Array(0, 0, -1, 1, 0, 1))
> 874 assert(dm1.numNonzeros === 3)
> 875 assert(dm1.numActives === 6)
> 876
> 877 val sm1 = Matrices.sparse(3, 2, Array(0, 2, 3), Array(0, 2, 1), Array(0.0, -1.2, 0.0))
> 878 assert(sm1.numNonzeros === 1)
> 879 assert(sm1.numActives === 3)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org