You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Udbhav Agrawal (JIRA)" <ji...@apache.org> on 2019/05/14 11:13:00 UTC
[jira] [Updated] (SPARK-27703) kubernetes jobs are failing with
Unsatisfiedlink error
[ https://issues.apache.org/jira/browse/SPARK-27703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Udbhav Agrawal updated SPARK-27703:
-----------------------------------
Summary: kubernetes jobs are failing with Unsatisfiedlink error (was: kubernetes jobs are filing with Unsatisfiedlink error)
> kubernetes jobs are failing with Unsatisfiedlink error
> ------------------------------------------------------
>
> Key: SPARK-27703
> URL: https://issues.apache.org/jira/browse/SPARK-27703
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.4.0
> Reporter: Udbhav Agrawal
> Priority: Major
>
> I encountered this issue while using ctas parquet queries in kubernetes cluster mode.
> this is the exception i encountered:
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
> at org.apache.spark.scheduler.Task.run(Task.scala:121)
> at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:403)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:409)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.UnsatisfiedLinkError: /tmp/[snappy-1.1.7-e963aa7c-4a30-4795-ad62-0a4d6a3c2a46-libsnappyjava.so|http://snappy-1.1.7-e963aa7c-4a30-4795-ad62-0a4d6a3c2a46-libsnappyjava.so/]: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /tmp/[snappy-1.1.7-e963aa7c-4a30-4795-ad62-0a4d6a3c2a46-libsnappyjava.so|http://snappy-1.1.7-e963aa7c-4a30-4795-ad62-0a4d6a3c2a46-libsnappyjava.so/])
> at java.lang.ClassLoader$NativeLibrary.load(Native Method)
> at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
> at java.lang.Runtime.load0(Runtime.java:809)
> at java.lang.System.load(System.java:1086)
> at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:179)
> at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:154)
> at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
> at parquet.hadoop.codec.SnappyCompressor.compress(SnappyCompressor.java:67)
> at org.apache.hadoop.io.compress.CompressorStream.compress(CompressorStream.java:81)
> at org.apache.hadoop.io.compress.CompressorStream.finish(CompressorStream.java:92)
> at parquet.hadoop.CodecFactory$BytesCompressor.compress(CodecFactory.java:112)
> at parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:89)
> at parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:153)
> at parquet.column.impl.ColumnWriterV1.accountForValueWritten(ColumnWriterV1.java:114)
> at parquet.column.impl.ColumnWriterV1.write(ColumnWriterV1.java:206)
> at parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.addBinary(MessageColumnIO.java:346)
> at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writePrimitive(DataWritableWriter.java:281)
> at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeValue(DataWritableWriter.java:106)
> at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeGroupFields(DataWritableWriter.java:89)
> at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:60)
> at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
> at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
> at parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:121)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
> at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
> at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
> at org.apache.spark.sql.hive.execution.HiveOutputWriter.write(HiveFileFormat.scala:149)
> at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.write(FileFormatDataWriter.scala:137)
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:245)
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
> at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
>
> Diggin in more i discovered after exec into docker image that particular so file is present in /bin64 folder and ld_config_path is not configured for that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org