You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2026/04/10 11:53:04 UTC
[PR] [SPARK-56411][SQL] SPARK-56411: Register Decimal in KryoSerializer so cached-batch spill works [spark]
LuciferYang opened a new pull request, #55287:
URL: https://github.com/apache/spark/pull/55287
### What changes were proposed in this pull request?
Register `Decimal`, `Decimal[]`, `java.math.BigDecimal`, and `java.math.BigInteger` in `KryoSerializer.loadableSparkClasses` so that Kryo strict registration mode(`spark.kryo.registrationRequired=true`) can serialize these types without throwing `Class is not registered`.
### Why are the changes needed?
`DefaultCachedBatchSerializer` writes cached batch stats via `kryo.writeClassAndObject(output, batch.stats)`. The stats row (`GenericInternalRow`) can contain `Decimal` values (e.g. min/max stats for a `DecimalType` column). When Kryo walks the row it encounters:
- `org.apache.spark.sql.types.Decimal` -- the value class itself.
- `java.math.BigDecimal` / `java.math.BigInteger` -- used internally when the value overflows `Long` precision (> 18 digits).
None of these were registered. Under strict mode, any cache spill or eviction of a batch with Decimal stats crashes:
```
com.esotericsoftware.kryo.KryoException: Class is not registered: org.apache.spark.sql.types.Decimal
```
This can be reproduced by caching a table with `DecimalType` columns(e.g. `CACHE TABLE store_sales`) and triggering a memory-pressure spill under Kryo strict mode.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Added new tests `DefaultCachedBatchKryoSerializerSuite`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56411][SQL] Register Decimal in KryoSerializer so cached-batch spill works [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang closed pull request #55287: [SPARK-56411][SQL] Register Decimal in KryoSerializer so cached-batch spill works
URL: https://github.com/apache/spark/pull/55287
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56411][SQL] Register Decimal in KryoSerializer so cached-batch spill works [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #55287:
URL: https://github.com/apache/spark/pull/55287#issuecomment-4231929241
Merged into master. Thanks @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56411][SQL] Register Decimal in KryoSerializer so cached-batch spill works [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #55287:
URL: https://github.com/apache/spark/pull/55287#issuecomment-4223627117
> Could you revise the PR title, @LuciferYang ?
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56411][SQL] SPARK-56411: Register Decimal in KryoSerializer so cached-batch spill works [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #55287:
URL: https://github.com/apache/spark/pull/55287#issuecomment-4223584408
Could you revise the PR title, @LuciferYang ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56411][SQL] Register Decimal in KryoSerializer so cached-batch spill works [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #55287:
URL: https://github.com/apache/spark/pull/55287#issuecomment-4223795625
Thank you @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org