You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Mridul Muralidharan (Jira)" <ji...@apache.org> on 2022/01/06 23:07:00 UTC
[jira] [Resolved] (SPARK-37710) Add detailed log message for java.io.IOException occurring on Kryo flow
[ https://issues.apache.org/jira/browse/SPARK-37710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mridul Muralidharan resolved SPARK-37710.
-----------------------------------------
Fix Version/s: 3.3.0
Resolution: Fixed
Issue resolved by pull request 34980
[https://github.com/apache/spark/pull/34980]
> Add detailed log message for java.io.IOException occurring on Kryo flow
> -----------------------------------------------------------------------
>
> Key: SPARK-37710
> URL: https://issues.apache.org/jira/browse/SPARK-37710
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.1.1
> Reporter: Eren Avsarogullari
> Assignee: Eren Avsarogullari
> Priority: Major
> Fix For: 3.3.0
>
>
> *Input/output error* usually points environmental issues such as disk read/write failures due to disk corruption, network access failures etc. This PR aims to be added detailed error message to catch this kind of environmental cases occurring on problematic BlockManager and logs with *BlockManager hostname, blockId and blockPath* details.
> Following stack-trace occurred on disk corruption:
> {code:java}
> com.esotericsoftware.kryo.KryoException: java.io.IOException: Input/output error
> Serialization trace:
> buffers (org.apache.spark.sql.execution.columnar.DefaultCachedBatch)
> at com.esotericsoftware.kryo.io.Input.fill(Input.java:166)
> at com.esotericsoftware.kryo.io.Input.require(Input.java:196)
> at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:346)
> at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:326)
> at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.read(DefaultArraySerializers.java:55)
> at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.read(DefaultArraySerializers.java:38)
> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:789)
> at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:381)
> at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:302)
> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:789)
> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:816)
> at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:296)
> at org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:168)
> at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
> at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
> at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
> at org.apache.spark.storage.BlockManager.maybeCacheDiskValuesInMemory(BlockManager.scala:1569)
> at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:877)
> at org.apache.spark.storage.BlockManager.get(BlockManager.scala:1163)
> ...
> Caused by: java.io.IOException: Input/output error
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> at net.jpountz.lz4.LZ4BlockInputStream.tryReadFully(LZ4BlockInputStream.java:269)
> at net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:280)
> at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:243)
> at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:157)
> at com.esotericsoftware.kryo.io.Input.fill(Input.java:164)
> ... 87 more {code}
> *Proposed Error Message:*
> {code:java}
> java.io.IOException: Input/output error. BlockManagerId(driver, localhost, 49455, None) - blockId: test_my-block-id - blockDiskPath: /private/var/folders/kj/mccyycwn6mjdwnglw9g3k6pm0000gq/T/blockmgr-12dba181-771e-4ff9-a2bc-fa3ce6dbabfa/11/test_my-block-id {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org