You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Marcelo Vanzin <va...@cloudera.com> on 2015/11/26 01:51:33 UTC
VerifyError running Spark SQL code?
I've been running into this error when running Spark SQL recently; no
matter what I try (completely clean build or anything else) doesn't
seem to fix it. Anyone has some idea of what's wrong?
[info] Exception encountered when attempting to run a suite with class
name: org.apache.spark.sql.execution.ui.SQLListenerMemoryLeakSuite ***
ABORTED *** (4 seconds, 111 milliseconds)
[info] java.lang.VerifyError: Bad <init> method call from inside of a branch
[info] Exception Details:
[info] Location:
[info] org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.<init>(Lorg/apache/spark/sql/catalyst/expressions/Expression;Lorg/apache/spark/sql/catalyst/expressions/Expression;)V
@82: invokespecial
Same happens with spark shell (when instantiating SQLContext), so not
an issue with the test code...
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: VerifyError running Spark SQL code?
Posted by Marcelo Vanzin <va...@cloudera.com>.
Seems to be some new thing with recent JDK updates according to the
intertubes. This patch seems to work around it:
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.scala
@@ -63,11 +63,7 @@ case class HyperLogLogPlusPlus(
def this(child: Expression, relativeSD: Expression) = {
this(
child = child,
- relativeSD = relativeSD match {
- case Literal(d: Double, DoubleType) => d
- case _ =>
- throw new AnalysisException("The second argument should be
a double literal.")
- },
+ relativeSD = HyperLogLogPlusPlus.validateRelativeSd(relativeSD),
mutableAggBufferOffset = 0,
inputAggBufferOffset = 0)
}
@@ -448,4 +444,11 @@ object HyperLogLogPlusPlus {
Array(189083, 185696.913, 182348.774, 179035.946, 175762.762,
172526.444, 169329.754, 166166.099, 163043.269, 159958.91, 156907.912,
153906.845,
150924.199, 147996.568, 145093.457, 142239.233, 139421.475, 136632.27,
133889.588, 131174.2, 128511.619, 125868.621, 123265.385, 120721.061,
118181.76
9, 115709.456, 113252.446, 110840.198, 108465.099, 106126.164,
103823.469, 101556.618, 99308.004, 97124.508, 94937.803, 92833.731,
90745.061, 88677.62
7, 86617.47, 84650.442, 82697.833, 80769.132, 78879.629, 77014.432,
75215.626, 73384.587, 71652.482, 69895.93, 68209.301, 66553.669,
64921.981, 63310.
323, 61742.115, 60205.018, 58698.658, 57190.657, 55760.865, 54331.169,
52908.167, 51550.273, 50225.254, 48922.421, 47614.533, 46362.049,
45098.569, 43
926.083, 42736.03, 41593.473, 40425.26, 39316.237, 38243.651,
37170.617, 36114.609, 35084.19, 34117.233, 33206.509, 32231.505,
31318.728, 30403.404, 2
9540.0550000001, 28679.236, 27825.862, 26965.216, 26179.148, 25462.08,
24645.952, 23922.523, 23198.144, 22529.128, 21762.4179999999,
21134.779, 20459.
117, 19840.818, 19187.04, 18636.3689999999, 17982.831,
17439.7389999999, 16874.547, 16358.2169999999, 15835.684, 15352.914,
14823.681, 14329.313, 1381
6.897, 13342.874, 12880.882, 12491.648, 12021.254, 11625.392,
11293.7610000001, 10813.697, 10456.209, 10099.074, 9755.39000000001,
9393.18500000006, 9
047.57900000003, 8657.98499999999, 8395.85900000005, 8033,
7736.95900000003, 7430.59699999995, 7258.47699999996,
6924.58200000005, 6691.29399999999, 6
357.92500000005, 6202.05700000003, 5921.19700000004, 5628.28399999999,
5404.96799999999, 5226.71100000001, 4990.75600000005,
4799.77399999998, 4622.93
099999998, 4472.478, 4171.78700000001, 3957.46299999999,
3868.95200000005, 3691.14300000004, 3474.63100000005,
3341.67200000002, 3109.14000000001, 307
1.97400000005, 2796.40399999998, 2756.17799999996, 2611.46999999997,
2471.93000000005, 2382.26399999997, 2209.22400000005,
2142.28399999999, 2013.9610
0000001, 1911.18999999994, 1818.27099999995, 1668.47900000005,
1519.65800000005, 1469.67599999998, 1367.13800000004,
1248.52899999998, 1181.2360000000
3, 1022.71900000004, 1088.20700000005, 959.03600000008,
876.095999999903, 791.183999999892, 703.337000000058,
731.949999999953, 586.86400000006, 526.0
24999999907, 323.004999999888, 320.448000000091, 340.672999999952,
309.638999999966, 216.601999999955, 102.922999999952,
19.2399999999907, -0.11400000
0059605, -32.6240000000689, -89.3179999999702, -153.497999999905,
-64.2970000000205, -143.695999999996, -259.497999999905,
-253.017999999924, -213.948
000000091, -397.590000000084, -434.006000000052, -403.475000000093,
-297.958000000101, -404.317000000039, -528.898999999976,
-506.621000000043, -513.2
05000000075, -479.351000000024, -596.139999999898, -527.016999999993,
-664.681000000099, -680.306000000099, -704.050000000047,
-850.486000000034, -757
.43200000003, -713.308999999892)
)
// scalastyle:on
+
+ private def validateRelativeSd(relativeSD: Expression): Double =
relativeSD match {
+ case Literal(d: Double, DoubleType) => d
+ case _ =>
+ throw new AnalysisException("The second argument should be a
double literal.")
+ }
+
}
On Wed, Nov 25, 2015 at 5:29 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> $ java -version
> java version "1.7.0_67"
> Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
>
> (On Linux.) It's not that particular suite, though, it's anything I do
> that touches Spark SQL...
>
> On Wed, Nov 25, 2015 at 4:54 PM, Josh Rosen <jo...@databricks.com> wrote:
>> I think I've also seen this issue as well, but in a different suite. I
>> wasn't able to easily get to the bottom of it, though. What JDK / JRE are
>> you using? I'm on
>>
>>
>> Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
>> Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
>>
>> on OSX.
>>
>> On Wed, Nov 25, 2015 at 4:51 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
>>>
>>> I've been running into this error when running Spark SQL recently; no
>>> matter what I try (completely clean build or anything else) doesn't
>>> seem to fix it. Anyone has some idea of what's wrong?
>>>
>>> [info] Exception encountered when attempting to run a suite with class
>>> name: org.apache.spark.sql.execution.ui.SQLListenerMemoryLeakSuite ***
>>> ABORTED *** (4 seconds, 111 milliseconds)
>>> [info] java.lang.VerifyError: Bad <init> method call from inside of a
>>> branch
>>> [info] Exception Details:
>>> [info] Location:
>>> [info]
>>> org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.<init>(Lorg/apache/spark/sql/catalyst/expressions/Expression;Lorg/apache/spark/sql/catalyst/expressions/Expression;)V
>>> @82: invokespecial
>>>
>>> Same happens with spark shell (when instantiating SQLContext), so not
>>> an issue with the test code...
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>
>
>
>
> --
> Marcelo
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: VerifyError running Spark SQL code?
Posted by Marcelo Vanzin <va...@cloudera.com>.
$ java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
(On Linux.) It's not that particular suite, though, it's anything I do
that touches Spark SQL...
On Wed, Nov 25, 2015 at 4:54 PM, Josh Rosen <jo...@databricks.com> wrote:
> I think I've also seen this issue as well, but in a different suite. I
> wasn't able to easily get to the bottom of it, though. What JDK / JRE are
> you using? I'm on
>
>
> Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
>
> on OSX.
>
> On Wed, Nov 25, 2015 at 4:51 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
>>
>> I've been running into this error when running Spark SQL recently; no
>> matter what I try (completely clean build or anything else) doesn't
>> seem to fix it. Anyone has some idea of what's wrong?
>>
>> [info] Exception encountered when attempting to run a suite with class
>> name: org.apache.spark.sql.execution.ui.SQLListenerMemoryLeakSuite ***
>> ABORTED *** (4 seconds, 111 milliseconds)
>> [info] java.lang.VerifyError: Bad <init> method call from inside of a
>> branch
>> [info] Exception Details:
>> [info] Location:
>> [info]
>> org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.<init>(Lorg/apache/spark/sql/catalyst/expressions/Expression;Lorg/apache/spark/sql/catalyst/expressions/Expression;)V
>> @82: invokespecial
>>
>> Same happens with spark shell (when instantiating SQLContext), so not
>> an issue with the test code...
>>
>> --
>> Marcelo
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: VerifyError running Spark SQL code?
Posted by Josh Rosen <jo...@databricks.com>.
I think I've also seen this issue as well, but in a different suite. I
wasn't able to easily get to the bottom of it, though. What JDK / JRE are
you using? I'm on
Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
on OSX.
On Wed, Nov 25, 2015 at 4:51 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> I've been running into this error when running Spark SQL recently; no
> matter what I try (completely clean build or anything else) doesn't
> seem to fix it. Anyone has some idea of what's wrong?
>
> [info] Exception encountered when attempting to run a suite with class
> name: org.apache.spark.sql.execution.ui.SQLListenerMemoryLeakSuite ***
> ABORTED *** (4 seconds, 111 milliseconds)
> [info] java.lang.VerifyError: Bad <init> method call from inside of a
> branch
> [info] Exception Details:
> [info] Location:
> [info]
> org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.<init>(Lorg/apache/spark/sql/catalyst/expressions/Expression;Lorg/apache/spark/sql/catalyst/expressions/Expression;)V
> @82: invokespecial
>
> Same happens with spark shell (when instantiating SQLContext), so not
> an issue with the test code...
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>