You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Beam JIRA Bot (Jira)" <ji...@apache.org> on 2020/09/13 17:08:03 UTC
[jira] [Updated] (BEAM-10083) Spark Runner Tests failing [Java 11]
[ https://issues.apache.org/jira/browse/BEAM-10083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Beam JIRA Bot updated BEAM-10083:
---------------------------------
Labels: stale-P2 (was: )
> Spark Runner Tests failing [Java 11]
> ------------------------------------
>
> Key: BEAM-10083
> URL: https://issues.apache.org/jira/browse/BEAM-10083
> Project: Beam
> Issue Type: Sub-task
> Components: runner-spark
> Reporter: Pawel Pasterz
> Priority: P2
> Labels: stale-P2
>
> Gradle task *_:runners:spark:test_* fails during Java 11 Precommit job
>
> Example stack trace:
> {code:java}
> > Task :runners:spark:test
> 20/05/26 07:26:31 INFO org.apache.beam.runners.spark.metrics.MetricsAccumulator: Instantiated metrics accumulator: {
> "metrics": {
> }
> }
> org.apache.beam.runners.spark.structuredstreaming.StructuredStreamingPipelineStateTest > testBatchPipelineRunningState STANDARD_ERROR
> 20/05/26 07:26:32 INFO org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingRunner: *** SparkStructuredStreamingRunner is based on spark structured streaming framework and is no more
> based on RDD/DStream API. See
> https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
> It is still experimental, its coverage of the Beam model is partial. ***
> org.apache.beam.runners.spark.SparkPortableExecutionTest > testExecution STANDARD_ERROR
> 20/05/26 07:26:33 WARN org.apache.beam.runners.spark.translation.GroupNonMergingWindowsFunctions: Either coder LengthPrefixCoder(ByteArrayCoder) or GlobalWindow$Coder is not consistent with equals. That might cause issues on some runners.
> org.apache.beam.runners.spark.structuredstreaming.translation.batch.FlattenTest > testFlatten STANDARD_ERROR
> 20/05/26 07:26:34 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> org.apache.beam.runners.spark.SparkPortableExecutionTest > testExecution STANDARD_ERROR
> 20/05/26 07:26:34 ERROR org.apache.beam.runners.jobsubmission.JobInvocation: Error during job invocation fakeId.
> java.lang.IllegalArgumentException: Unsupported class file major version 55
> at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:166)
> at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:148)
> at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:136)
> at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:237)
> at org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:49)
> at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:517)
> at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:500)
> at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
> at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134)
> at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134)
> at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:134)
> at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
> at org.apache.spark.util.FieldAccessFinder$$anon$3.visitMethodInsn(ClosureCleaner.scala:500)
> at org.apache.xbean.asm6.ClassReader.readCode(ClassReader.java:2175)
> at org.apache.xbean.asm6.ClassReader.readMethod(ClassReader.java:1238)
> at org.apache.xbean.asm6.ClassReader.accept(ClassReader.java:631)
> at org.apache.xbean.asm6.ClassReader.accept(ClassReader.java:355)
> at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:307)
> at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:306)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:306)
> at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)
> at org.apache.spark.SparkContext.clean(SparkContext.scala:2326)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2100)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
> at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:990)
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
> at org.apache.spark.rdd.RDD.collect(RDD.scala:989)
> at org.apache.spark.api.java.JavaRDDLike$class.collect(JavaRDDLike.scala:361)
> at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
> at org.apache.beam.runners.spark.translation.BoundedDataset.getBytes(BoundedDataset.java:76)
> at org.apache.beam.runners.spark.translation.SparkBatchPortablePipelineTranslator.broadcastSideInput(SparkBatchPortablePipelineTranslator.java:354)
> at org.apache.beam.runners.spark.translation.SparkBatchPortablePipelineTranslator.broadcastSideInputs(SparkBatchPortablePipelineTranslator.java:338)
> at org.apache.beam.runners.spark.translation.SparkBatchPortablePipelineTranslator.translateExecutableStage(SparkBatchPortablePipelineTranslator.java:216)
> at org.apache.beam.runners.spark.translation.SparkBatchPortablePipelineTranslator.translate(SparkBatchPortablePipelineTranslator.java:138)
> at org.apache.beam.runners.spark.SparkPipelineRunner.lambda$run$1(SparkPipelineRunner.java:122)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> org.apache.beam.runners.spark.SparkPortableExecutionTest > testExecution FAILED
> java.lang.AssertionError: expected:<DONE> but was:<FAILED>
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.failNotEquals(Assert.java:835)
> at org.junit.Assert.assertEquals(Assert.java:120)
> at org.junit.Assert.assertEquals(Assert.java:146)
> at org.apache.beam.runners.spark.SparkPortableExecutionTest.testExecution(SparkPortableExecutionTest.java:159)
> org.apache.beam.runners.spark.SparkPortableExecutionTest > testExecStageWithMultipleOutputs STANDARD_ERROR
> 20/05/26 07:26:35 INFO org.apache.beam.runners.jobsubmission.JobInvocation: Starting job invocation testExecStageWithMultipleOutputs
> 20/05/26 07:26:36 INFO org.apache.beam.runners.spark.SparkPipelineRunner: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath
> 20/05/26 07:26:36 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Will stage 289 files. (Enable logging at DEBUG level to see which files will be staged.)
> 20/05/26 07:26:36 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Running job testExecStageWithMultipleOutputs on Spark master local[4]
> 20/05/26 07:26:37 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job testExecStageWithMultipleOutputs: Pipeline translated successfully. Computing outputs
> Gradle Test Executor 114 started executing tests.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)