You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zhengchenyu (via GitHub)" <gi...@apache.org> on 2024/07/17 06:23:18 UTC
[PR] [#1745] feat(remote merge): Introduce a common serializer. [incubator-uniffle]
zhengchenyu opened a new pull request, #1916:
URL: https://github.com/apache/incubator-uniffle/pull/1916
### What changes were proposed in this pull request?
### Why are the changes needed?
Since merge needs to be performed on the ShuffleServer side, and serialization/deserialization may differ between versions of the compute framework. To avoid server-side processing of different versions of serialization/deserialization, unified serializer that is independent of the computing framework needs to be extracted.
Two types of serializers are extracted here: (1) Writable (2) Kryo. Writable serialization is used for classes that handle the org.apache.hadoop.io.Writable interface, used in the MR and TEZ frameworks. Kryo can serialize most classes and is generally used in the Spark framework.
Fix: # 1745
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
unit test and test in cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1745] feat(remote merge): Introduce a common serializer. [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1916:
URL: https://github.com/apache/incubator-uniffle/pull/1916#issuecomment-2232903854
Thanks for pushing this feature forward.
> Note: KryoSerializer will be introduced when spark is supported.
Could you help describe more that why spark don't support kryo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1745] feat(remote merge): Introduce a common serializer. [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1916:
URL: https://github.com/apache/incubator-uniffle/pull/1916#issuecomment-2232528438
Maybe we can remove the related implement of `kryo`. We don't need to support Spark at first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1745] feat(remote merge): Introduce a common serializer. [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1916:
URL: https://github.com/apache/incubator-uniffle/pull/1916#issuecomment-2232721541
Could you update your description?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1745] feat(remote merge): Introduce a common serializer. [incubator-uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1916:
URL: https://github.com/apache/incubator-uniffle/pull/1916#issuecomment-2232705171
## Test Results
2 702 files + 45 2 702 suites +45 5h 36m 25s :stopwatch: + 2m 53s
960 tests + 14 959 :white_check_mark: + 15 1 :zzz: ±0 0 :x: - 1
12 009 runs +210 11 994 :white_check_mark: +211 15 :zzz: ±0 0 :x: - 1
Results for commit 8e9f2915. ± Comparison against base commit f984efb0.
[test-results]:data:application/gzip;base64,H4sIADd9l2YC/03MTQ7CIBCG4as0rF0wYymMlzGUloTYH0NhZby7YKBl+T0M74dZt8wHe3QoOd46dkQX2j1Fr4PbtywclEiU3kO+oIHX9TyiMZkEXfRy70RwgtVuSXB9mr3ffREft9wE5JzKrFEAov60UhUVmup/t1Gzr6sLaTA1k0UCIZRCo0n2kmAcLRGOUiI3936QGsTEvj9E4sCOEgEAAA==
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1745] feat(remote merge): Introduce a common serializer. [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1916:
URL: https://github.com/apache/incubator-uniffle/pull/1916#discussion_r1680483444
##########
common/src/main/java/org/apache/uniffle/common/config/RssBaseConf.java:
##########
@@ -278,6 +280,31 @@ public class RssBaseConf extends RssConf {
.defaultValue(16)
.withDescription("start server service max retry");
+ /* Serialization */
Review Comment:
Could you add a document for these config options?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1745] feat(remote merge): Introduce a common serializer. [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1916:
URL: https://github.com/apache/incubator-uniffle/pull/1916#discussion_r1680771927
##########
common/src/main/java/org/apache/uniffle/common/config/RssBaseConf.java:
##########
@@ -278,6 +280,31 @@ public class RssBaseConf extends RssConf {
.defaultValue(16)
.withDescription("start server service max retry");
+ /* Serialization */
Review Comment:
If this could be specified in client side, it wil be better. Anyway, this is an initial PR, now this is good for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1745] feat(remote merge): Introduce a common serializer. [incubator-uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #1916:
URL: https://github.com/apache/incubator-uniffle/pull/1916#issuecomment-2232950095
@zuston
Tez and MR have been thoroughly tested on our internal clusters using production job. However, spark has not been tested with enough production job.
There are still two issues to be resolved for spark.
* (1) Spark SQL may still spill to disk.
Some aggregators assume that the records passed to them are not sorted, so may need large memory, then may spill to disk.
For Hive, aggerator assumes that the records passed are sorted, so they can be aggregated in memory.
For sparksql, we need to ensure that the shuffle is sorted and provide an in-memory the aggregator while supporting merge sort.
* (2) For remote merge, the records within a block need to be sorted.
Spark's WriteBufferManager does not require sorted record within a block, but remote merge need. WriteBufferManager need to support sort record within a block.
As the two reason, spark related codes will not be submitted for now. The kryo is only used for spark. When submitting spark-related code, support KryoSerializer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1745] feat(remote merge): Introduce a common serializer. [incubator-uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #1916:
URL: https://github.com/apache/incubator-uniffle/pull/1916#issuecomment-2232547659
> Maybe we can remove the related implement of `kryo`. We don't need to support Spark at first.
OK! Since spark is not supported firstly, I will remove all kryo related first. Add the kryo serializer until spark is supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1745] feat(remote merge): Introduce a common serializer. [incubator-uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on code in PR #1916:
URL: https://github.com/apache/incubator-uniffle/pull/1916#discussion_r1680523583
##########
common/src/main/java/org/apache/uniffle/common/config/RssBaseConf.java:
##########
@@ -278,6 +280,31 @@ public class RssBaseConf extends RssConf {
.defaultValue(16)
.withDescription("start server service max retry");
+ /* Serialization */
Review Comment:
After remove kryo, only config 'rss.io.serializations' remained. For now, only one value here. Users do not have to configure it. So I don't think it is necessary to add client_guide.md or server_guide.md temporarily.
For remote merge, some parameters are indeed introduced, I want to create a separate PR to describe it.
For new users, just need to enable this feature by the one config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1745] feat(remote merge): Introduce a common serializer. [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1916:
URL: https://github.com/apache/incubator-uniffle/pull/1916#issuecomment-2233051526
> @zuston Tez and MR have been thoroughly tested on our internal clusters using production job. However, spark has not been tested with enough production job.
>
> There are still two issues to be resolved for spark.
>
> * (1) Spark SQL may still spill to disk.
> Some aggregators assume that the records passed to them are not sorted, so may need large memory, then may spill to disk.
> For Hive, aggerator assumes that the records passed are sorted, so they can be aggregated in memory.
> For sparksql, we need to ensure that the shuffle is sorted and provide an in-memory the aggregator while supporting merge sort.
> * (2) For remote merge, the records within a block need to be sorted.
> Spark's WriteBufferManager does not require sorted record within a block, but remote merge need. WriteBufferManager need to support sort record within a block.
>
> As the two reason, spark related codes will not be submitted for now. The kryo is only used for spark, then KryoSerializer will not be submitted for now. When submitting spark-related code, will support KryoSerializer.
Got it. Thanks for your explaination
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1745] feat(remote merge): Introduce a common serializer. [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi merged PR #1916:
URL: https://github.com/apache/incubator-uniffle/pull/1916
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1745] feat(remote merge): Introduce a common serializer. [incubator-uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #1916:
URL: https://github.com/apache/incubator-uniffle/pull/1916#issuecomment-2232525804
This PR is the first work #1239, See #1239 for detailed documentation. See https://github.com/apache/incubator-uniffle/pull/1906 for the complete code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org