You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/11/19 15:26:56 UTC
[GitHub] [incubator-uniffle] jerqi opened a new issue, #342: [Improvement] Check Spark Serializer type
jerqi opened a new issue, #342:
URL: https://github.com/apache/incubator-uniffle/issues/342
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
Spark have multiple serializers. We support the spark serializer which supportsRelocationOfSerializedObjects.
You can see https://github.com/apache/spark/blob/25849684b78cca6651e25d6efc9644a576e7e20f/core/src/main/scala/org/apache/spark/serializer/Serializer.scala#L98
Spark have three kinds of serializer
org.apache.spark.serializer.JavaSerializer
org.apache.spark.sql.execution.UnsafeRowSerializer
org.apache.spark.serializer.KryoSerializer
Only org.apache.spark.serializer.JavaSerializer don't support RelocationOfSerializedObjects.
So when we find the parameters to use `org.apache.spark.serializer.JavaSerializer`, We should throw an exception, similar code that you can refer to https://github.com/apache/incubator-uniffle/blob/d09b40b90b8db613dbd346e0cd91f70f3950b430/client-spark/spark2/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java#L141
### How should we improve?
check Serializer whether support supportsRelocationOfSerializedObjects, if not, throw an exception.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi closed issue #342: [Improvement] Check Spark Serializer type
Posted by GitBox <gi...@apache.org>.
jerqi closed issue #342: [Improvement] Check Spark Serializer type
URL: https://github.com/apache/incubator-uniffle/issues/342
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org