You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cxzl25 (via GitHub)" <gi...@apache.org> on 2025/11/24 12:19:52 UTC
[PR] [SPARK-54489][CORE] Avoid returning corrupted Kryo objects [spark]
cxzl25 opened a new pull request, #53193:
URL: https://github.com/apache/spark/pull/53193
### What changes were proposed in this pull request?
When `Kryo.writeClassAndObject` throws a `NegativeArraySizeException`, the Kryo instance shall no longer be returned to the pool.
### Why are the changes needed?
`https://github.com/EsotericSoftware/kryo/pull/1198`
When Kryo.writeClassAndObject fails to resize, it throws a `NegativeArraySizeException`.
Spark then returns this Kryo instance to the pool. The next time this object is reused, it will always throw an `ArrayIndexOutOfBoundsException`.
```java
java.lang.NegativeArraySizeException: -2147483645
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:542)
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.putStash(IdentityObjectIntMap.java:306)
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.push(IdentityObjectIntMap.java:300)
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:162)
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.putStash(IdentityObjectIntMap.java:307)
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.push(IdentityObjectIntMap.java:300)
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:162)
at com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapReferenceResolver.java:41)
at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:681)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:646)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:361)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:302)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
at org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:269)
at org.apache.spark.broadcast.TorrentBroadcast$.$anonfun$blockifyObject$4(TorrentBroadcast.scala:358)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1510)
at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:360)
```
```java
java.lang.ArrayIndexOutOfBoundsException: Index 688291177 out of bounds for length 3
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.get(IdentityObjectIntMap.java:322)
at com.esotericsoftware.kryo.util.MapReferenceResolver.getWrittenId(MapReferenceResolver.java:46)
at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:671)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:646)
at org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:269)
at org.apache.spark.broadcast.TorrentBroadcast$.$anonfun$blockifyObject$4(TorrentBroadcast.scala:358)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1510)
at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:360)
```
Simple test
```java
IdentityObjectIntMap identityObjectIntMap = new IdentityObjectIntMap(1073741824, 0.8f);
try {
identityObjectIntMap.put("k1", 1);
identityObjectIntMap.clear((1073741824) << 1); // Simulate resize
} catch (NegativeArraySizeException e) {
e.printStackTrace(); // Expected
}
identityObjectIntMap.clear(2048);
identityObjectIntMap.put("k1", 1); // ArrayIndexOutOfBoundsException
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54489][CORE] Avoid returning corrupted Kryo objects [spark]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on PR #53193:
URL: https://github.com/apache/spark/pull/53193#issuecomment-3619846743
hmm, I cant seem to reproduce it with the test you have provided @cxzl25 ...
```
$ ./bin/spark-shell --driver-memory 64g
<snip>
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.1.0-SNAPSHOT
/_/
Using Scala version 2.13.16 (OpenJDK 64-Bit Server VM, Java 17.0.17)
<snip>
Spark session available as 'spark'.
scala> import com.esotericsoftware.kryo.util._
import com.esotericsoftware.kryo.util._
scala> val identityObjectIntMap = new IdentityObjectIntMap[String](1073741824, 0.8f)
val identityObjectIntMap: com.esotericsoftware.kryo.util.IdentityObjectIntMap[String] = {}
scala> identityObjectIntMap.put("k1", 1)
scala> identityObjectIntMap.clear((1073741824) << 1)
java.lang.NegativeArraySizeException: -2147483645
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:542)
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.clear(IdentityObjectIntMap.java:461)
... 42 elided
scala> identityObjectIntMap.clear(2048)
scala> identityObjectIntMap.put("k1", 1)
scala>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54489][CORE] Avoid returning corrupted Kryo objects [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #53193:
URL: https://github.com/apache/spark/pull/53193#issuecomment-3572883147
Is it possible to add a test here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54489][CORE] Avoid returning corrupted Kryo objects [spark]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on PR #53193:
URL: https://github.com/apache/spark/pull/53193#issuecomment-3621949879
> We do a reset, which should have cleared this map - still an issue ?
The clear method will only set the value of the keyTable array to null, but will not reset the keyTable, so the next put will trigger this problem.
com.esotericsoftware.kryo.util.IdentityObjectIntMap#clear(int)
```java
public void clear (int maximumCapacity) {
if (capacity <= maximumCapacity) {
clear();
return;
}
public void clear () {
K[] keyTable = this.keyTable;
for (int i = capacity + stashSize; i-- > 0;)
keyTable[i] = null;
size = 0;
stashSize = 0;
}
```
> I cant seem to reproduce it with the test you have provided
Because in `IdentityObjectIntMap`, it uses `System.identityHashCode` to calculate hash, so different environments may get different results.
I modified the reproduced code. Maybe this is easier to reproduce.
```scala
import com.esotericsoftware.kryo.util._
val identityObjectIntMap = new IdentityObjectIntMap[String](1073741824, 0.8f)
identityObjectIntMap.put("k1", 1)
identityObjectIntMap.clear((1073741824) << 1)
identityObjectIntMap.clear(2048)
for (i <- 0 until 10000) {
val s = new String("k_" + i)
val i1 = System.identityHashCode(s) & 2147483647
if (i1 > 1073741866) {
println(s"Found one: $s, hash: $i1")
identityObjectIntMap.put(s, 1)
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54489][CORE] Avoid returning corrupted Kryo objects [spark]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on PR #53193:
URL: https://github.com/apache/spark/pull/53193#issuecomment-3619836425
We do a reset, which should have cleared this map - still an issue ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org