[GitHub] [spark] mridulm edited a comment on pull request #34064: [SPARK-36820][SQL][test-hadoop2.7] Disable LZ4 test for Hadoop 2.7

[GitBox <gi...@apache.org>]

mridulm edited a comment on pull request #34064:
URL: https://github.com/apache/spark/pull/34064#issuecomment-924534422


   I am not sure if this Suite is also having issues with hadoop-2.7 profile; but the issue I had observed was in FileSuite in core.
   This was working fine in 3.1 IIRC
   Full failure message:
   ```
   - SequenceFile (compressed) - lz4 *** FAILED ***
     org.apache.spark.SparkException: Job aborted.
     at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:106)
     at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsHadoopDataset$1(PairRDDFunctions.scala:1090)
     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
     at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
     at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1088)
     at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsHadoopFile$4(PairRDDFunctions.scala:1061)
     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
     ...
     Cause: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1) (localhost executor driver): java.lang.RuntimeException: native lz4 library not available
           at org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
           at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
           at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
           at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1263)
           at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1156)
           at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1511)
           at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:277)
           at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
           at org.apache.hadoop.mapred.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:64)
           at org.apache.spark.internal.io.HadoopMapRedWriteConfigUtil.initWriter(SparkHadoopWriter.scala:238)
           at org.apache.spark.internal.io.SparkHadoopWriter$.executeTask(SparkHadoopWriter.scala:126)
           at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$write$1(SparkHadoopWriter.scala:88)
           at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
           at org.apache.spark.scheduler.Task.run(Task.scala:131)
           at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
           at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
           at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
   
   Driver stacktrace:
     at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2403)
     at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2352)
     at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2351)
     at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
     at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
     at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
     at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2351)
     at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1109)
     at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1109)
     at scala.Option.foreach(Option.scala:407)
     ...
     Cause: java.lang.RuntimeException: native lz4 library not available
     at org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
     at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
     at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
     at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1263)
     at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1156)
     at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1511)
     at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:277)
     at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
     at org.apache.hadoop.mapred.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:64)
     at org.apache.spark.internal.io.HadoopMapRedWriteConfigUtil.initWriter(SparkHadoopWriter.scala:238)
     ...
   
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org