You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/08/06 02:49:35 UTC
[GitHub] [incubator-uniffle] jerqi opened a new issue, #133: [Improvement] Netty replace Grpc on data transfer
jerqi opened a new issue, #133:
URL: https://github.com/apache/incubator-uniffle/issues/133
When we use Grpc, we find that our bottleneck is on Grpc, Grpc brings the cost of data copy and data serialization. We must encounter GC problems when we use Grpc. We should use Netty replace Grpc on data transfer and use off heap memory to reduce GC time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #133: [Umbrella] Netty replace Grpc on data transfer
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1501322678
I have some questions regarding the memory management of the shuffle server in the Netty implementation. Specifically, I'm curious about how we store shuffle data in memory. Do we use Netty's ByteBuf or a customized memory pool managed by Uniffle?
Based on my reading of the design document, there are two memory sections allocated for network and shuffle data storage. Would it be accurate to describe them as shared buffers?
I believe that introducing a customized memory pool could potentially address the above issues. Furthermore, this could also benefit the transfer of shuffle data using GRPC.
cc @leixm @advancedxy @jerqi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1427948689
Or we can write some test codes. After ShuffleServer receives the data, it does not perform disk writing operations, and verifies the throughput performance of ShuffleServer in the scenario where the impact of disk writing is excluded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1430654933
> The performance of disk is limited. Every HDD is 100MB/s. Every SDD is 1GB/s. Usually we test this with 1TB tpcds and 1TB Terasort. If we can get performance improve, we can merge some prs first.
How many ShuffleServer nodes were there in the previous test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1291441262
Yes, I'm free at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1287011268
This poc is based on Uber RSS netty implement, it's better that we can compare the Ali's rss netty implement, Uber's rss netty implement and other RSS netty implement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1291557185
> Yes, I'm free at this time.
Meeting link is https://meeting.tencent.com/dm/oR95wASCNe91
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1425103269
> After offline discussion, we should involve more people to participate in this issue. @zuston @leixm @advancedxy @kaijchen @smallzhongfeng @xianjingfeng Maybe we should have a regular meeting to discuss this issue continuously. WDYT?
It is ok for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1427910134
@leixm Have you do some performance tests?
@advancedxy @Gustfh @smallzhongfeng @zuston @leixm @kaijchen How about every wednesday 10:15 am every week for this issue regular meeting?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1259294828
After applying the POC of netty, will the problem of #230 exist? @jerqi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1256892365
Could we use grpc zero-copy to solve this problem? Alluxio also use Grpc to transfer data, I think the problem we encountered also exist in Alluxio
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1336743468
> > > 1.I did some POC tests, which can significantly reduce GCTime, I will sort out and upload some test results later 2.I made some comparisons. The design we propose is similar to Alibaba RSS. Do you have a better suggestion? I tried to do some tests on each RSS, but I found it difficult to do a specific performance comparison 3.read buffer refers to the memory usage of ShuffleServer, not the client
> >
> >
> > Could we have analysis about the every RSS's network transfer? How about poc code which I provide? Do they have the same performance?
>
> Any suggestions to measure the performance precisely?
I don't know whether micro benchmark can solve this problem. GC time can only be evaluted when we use several jobs to verify them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1287007372
Commit ID: 4463b80deacbcd47ddc151f8d7750d1e5f035077
git apply patch1
[netty1.txt](https://github.com/apache/incubator-uniffle/files/9839621/netty1.txt)
git apply patch2
[netty2.txt](https://github.com/apache/incubator-uniffle/files/9839628/netty2.txt)
After apply the patches, we can get the code
https://github.com/jerqi/incubator-uniffle/tree/netty_poc
According the poc test, we can get 10% performance improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1425277668
> After offline discussion, we should involve more people to participate in this issue. @zuston @leixm @advancedxy @kaijchen @smallzhongfeng @xianjingfeng Maybe we should have a regular meeting to discuss this issue continuously. WDYT?
It is ok for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1430660019
I will do tpcds test to verify the performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1424046227
> addressed
It is still in the development process and has not been applied to the production environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1424051399
> Can I add some ISSUE? Let's get involved. @jerqi
Yes, you can.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1425313049
> After offline discussion, we should involve more people to participate in this issue. @zuston @leixm @advancedxy @kaijchen @smallzhongfeng @xianjingfeng Maybe we should have a regular meeting to discuss this issue continuously. WDYT?
sounds good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1432915956
@leixm I think we can submit some small prs to our master branch first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1292964244
Offline Discussion Result:
We will compare the rpc frameworks of other rss, and @leixm will give a design doc. Next step, we will make the shuffle server will off heap as possible as we can.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1427946019
@jerqi Or do you have a better test method to accurately verify the throughput performance of ShuffleServer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Umbrella] Netty replace Grpc on data transfer
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1502617735
> > I believe we should do it in baby steps, the netty rpc framework should be introduced first, then the off-heap memory management, then unified memory management and spill.
>
> Yes. Let's reserve this
>
> > I haven't thought it throughly, but I believe we can learn from the best.
>
> https://github.com/flink-extended/flink-remote-shuffle is a good example
If you think it's worth doing, you should set up an issue to track it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Umbrella] Netty replace Grpc on data transfer
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1567223778
> How far off is this umbrella feature? I would be interested in taking a look at it as I have faced problems relating to large GC when running large shuffle jobs
Maybe this is helpful for you. https://github.com/apache/incubator-uniffle/issues/674#issuecomment-1454611218
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Umbrella] Netty replace Grpc on data transfer
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1564381193
> How far off is this umbrella feature? I would be interested in taking a look at it as I have faced problems relating to large GC when running large shuffle jobs.
Only two prs. But we should fix some issues. Maybe we can finish this feature at the end of June. What's your company? Do you use Uniffle in the production environment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1424046583
> Hi, @leixm xianming, has this issue already been addressed in your internal prod env? Do you any ETA for the PR to be uploaded?
It is still in the development process and has not been applied to the production environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1425371562
1. @leixm If we use Netty, we can transfer local file with `sendfile` that a zero copy linux api. We can remove read memory.
It's an important improvement.
2. And maybe we need some benchmark tests. We could set up our benchmark test systems. @LuciferYang Do you give us some advice about benchmark tests?
3. Reader and writer also need to support to use off heap memory to compress and decompress data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1425407089
> > Good idea, but the current performance bottleneck is mainly in ShuffleServer, we can improve the throughput of ShuffleServer first.
>
> You misunderstand me. When shuffle server read the local file, we use `sendfile` api to improve our shuffle server performance.
You are right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1286648067
> > What is the progress of this proposal, maybe I can join the development?
>
> We only have some POC code. I don't have much time, if you like , I can assign this issue to you. And I can provide our POC code to you.
+1. Could u help make this POC code to another branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1286556123
What is the progress of this proposal, maybe I can join the development?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1290123397
> @leixm @zuston Do you want to discuss this issue through a meeting? I will start a meeting to discuss the issue #80, I want to discuss this issue, too. There are some other issues which we need to discuss, so I will send a email to our dev mail list, and select a proper date to start the meeting. You can tell me what time you are free by the email.
Let's discuss it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1336715473
1.I did some POC tests, which can significantly reduce GCTime, I will sort out and upload some test results later
2.I made some comparisons. The design we propose is similar to Alibaba RSS. Do you have a better suggestion? I tried to do some tests on each RSS, but I found it difficult to do a specific performance comparison
3.read buffer refers to the memory usage of ShuffleServer, not the client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1336383847
@leixm Could you give us the authority of the comment for your design doc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1258888347
We have had a POC code about Netty. It will improve 10% performance for Uniffle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1336725283
> > 1.I did some POC tests, which can significantly reduce GCTime, I will sort out and upload some test results later 2.I made some comparisons. The design we propose is similar to Alibaba RSS. Do you have a better suggestion? I tried to do some tests on each RSS, but I found it difficult to do a specific performance comparison 3.read buffer refers to the memory usage of ShuffleServer, not the client
>
> Could we have analysis about the every RSS's network transfer? How about poc code which I provide? Do they have the same performance?
Any suggestions to measure the performance precisely?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1336717631
These RSS have one thing in common, using off-heap memory, their GC time is significantly smaller than Uniffle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1345076332
cc@jerqi I've included some test data in the design doc, including uniffle, celeborn, bytedance css, uber rss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1345108313
> cc@jerqi I've included some test data in the design doc, including uniffle, celeborn, bytedance css, uber rss.
If we can prove the Celeborn have better performance than Uber RSS. We can continue this our design. We also need to consider that we need use offheap memory to compress/decompress data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1288354232
Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1290293470
> > @leixm @zuston Do you want to discuss this issue through a meeting? I will start a meeting to discuss the issue #80, I want to discuss this issue, too. There are some other issues which we need to discuss, so I will send a email to our dev mail list, and select a proper date to start the meeting. You can tell me what time you are free by the email.
>
> Let's discuss it.
+1. Could we have a discussion in this week?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi closed issue #133: [Umbrella] Netty replace Grpc on data transfer
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi closed issue #133: [Umbrella] Netty replace Grpc on data transfer
URL: https://github.com/apache/incubator-uniffle/issues/133
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1291438611
@leixm I have already send an email https://lists.apache.org/thread/2jlm3fswmsxy619ldyo4px700p3ybnvc. Do you have time at 11 am (UTC +8) Thursday this week?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1336364293
https://docs.google.com/document/d/1srlBlLpJ3hbzd8ru5QaY5aCSLR9M2ttHXmlvcFC-SLg/edit?usp=sharing @jerqi @zuston @smallzhongfeng @smallzhongfeng Can you help review this design.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1336743806
> > @leixm Could you give us the authority of the comment for your design doc?
>
> done, Can you comment now?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1336724154
> @leixm Could you give us the authority of the comment for your design doc?
done, Can you comment now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1258899974
> We have had a POC code about Netty. It will improve 10% performance for Uniffle.
Sounds great! If I have time, I will do a zero-copy POC to compare with netty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1425399603
Good idea, but the current performance bottleneck is mainly in ShuffleServer, we can improve the throughput of ShuffleServer first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] kaijchen commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1425247639
Let's create a new branch such as `netty` for this feature. So we can collaborate on it before it's ready to be merged into master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] connorlwilkes commented on issue #133: [Umbrella] Netty replace Grpc on data transfer
Posted by "connorlwilkes (via GitHub)" <gi...@apache.org>.
connorlwilkes commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1564317385
How far off is this umbrella feature? I would be interested in taking a look at it as I have faced problems relating to large GC when running large shuffle jobs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi closed issue #133: [Umbrella] Netty replace Grpc on data transfer
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi closed issue #133: [Umbrella] Netty replace Grpc on data transfer
URL: https://github.com/apache/incubator-uniffle/issues/133
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on issue #133: [Umbrella] Netty replace Grpc on data transfer
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1502606303
> Based on my reading of the design document, there are two memory sections allocated for network and shuffle data storage. Would it be accurate to describe them as shared buffers?
It might be hard to share memory between shuffle data and network transport, but it's possible they are shared in a unified ByteBuffer pool, that means buffers should be released to pool then usable for other uses.
I believe we should do it in baby steps, the netty rpc framework should be introduced first, then the off-heap memory management, then unified memory management and spill. I haven't thought it throughly, but I believe we can learn from the best.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1288354383
> Commit ID: [4463b80](https://github.com/apache/incubator-uniffle/commit/4463b80deacbcd47ddc151f8d7750d1e5f035077) git apply patch1 [netty1.txt](https://github.com/apache/incubator-uniffle/files/9839621/netty1.txt)
>
> git apply patch2 [netty2.txt](https://github.com/apache/incubator-uniffle/files/9839628/netty2.txt)
>
> After apply the patches, we can get the code https://github.com/jerqi/incubator-uniffle/tree/netty_poc
>
> According the poc test, we can get 10% performance improvement.
Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1258885945
I dont look through the full design of GRPC zero-copy design and Alluxio's implementation.
Maybe we could do the POC to test the performance compared with the netty implementation. Overall, it looks easy to extend the zero-copy by GRPC, like this https://github.com/GoogleCloudPlatform/grpc-gcp-java/pull/77
From my perspective, GRPC is the general and cross-language, that means it make possible to implement the shuffle-server by other languages like rust to avoid GC and obtain better memory management in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1425279519
> After offline discussion, we should involve more people to participate in this issue. @zuston @leixm @advancedxy @kaijchen @smallzhongfeng @xianjingfeng Maybe we should have a regular meeting to discuss this issue continuously. WDYT?
+1. Looking forward this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1425416699
> 1. @leixm If we use Netty, we can transfer local file with `sendfile` that a zero copy linux api. We can remove read memory.
> It's an important improvement.
> 2. And maybe we need some benchmark tests. We could set up our benchmark test systems. @LuciferYang Do you give us some advice about benchmark tests?
> 3. Reader and writer also need to support to use off heap memory to compress and decompress data.
These ideas are very good, we can raise issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1425406515
You are right.
> > Good idea, but the current performance bottleneck is mainly in ShuffleServer, we can improve the throughput of ShuffleServer first.
>
> You misunderstand me. When shuffle server read the local file, we use `sendfile` api to improve our shuffle server performance.
you are right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1424044217
@advancedxy @jerqi I have uploaded a POC code, which is helpful to alleviate the problem of FullGC, but many places still need to be optimized. https://github.com/leixm/incubator-uniffle/tree/netty_poc_2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1424048330
Can I add some ISSUE? Let's get involved. @jerqi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1424198644
> https://github.com/leixm/incubator-uniffle/tree/netty_poc_2
Great. Do you think is it possible to submit an WIP pr now?
> It is still in the development process and has not been applied to the production environment.
No problem, let's create issues/prs targets your branch and discuss more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] smallzhongfeng commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1425244146
> After offline discussion, we should involve more people to participate in this issue. @zuston @leixm @advancedxy @kaijchen @smallzhongfeng @xianjingfeng Maybe we should have a regular meeting to discuss this issue continuously. WDYT?
It is ok for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1427944575
I am doing some tests on the throughput performance of ShuffleServer, but the comparison results are not very good. I found that the performance of ShuffleServer has a lot to do with the disk write performance, and it needs to be better when the disk throughput performance is greater than the ShuffleServer throughput performance. Verify whether the throughput performance of ShuffleServer has improved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1428992705
The performance of disk is limited. Every HDD is 100MB/s. Every SDD is 1GB/s.
Usually we test this with 1TB tpcds and 1TB Terasort. If we can get performance improve, we can merge some prs first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1286644879
> What is the progress of this proposal, maybe I can join the development?
We only have some POC code. I don't have much time, if you like , I can assign this issue to you. And I can provide our POC code to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1336722122
> 1.I did some POC tests, which can significantly reduce GCTime, I will sort out and upload some test results later 2.I made some comparisons. The design we propose is similar to Alibaba RSS. Do you have a better suggestion? I tried to do some tests on each RSS, but I found it difficult to do a specific performance comparison 3.read buffer refers to the memory usage of ShuffleServer, not the client
Could we have analysis about the every RSS's network transfer? How about poc code which I provide? Do they have the same performance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1336728787
I think we can evaluate whether the performance has improved in two parts. The first part is GC time, which is relatively simple. You can evaluate it by viewing the process through the running time of the app and jstat. The second part is whether serialization/deserialization has improved , this is more complicated. At present, what I think of is to evaluate through transportTime, which includes the serialization/deserialization time of the client, the network transmission time, and the serialization/deserialization time of ShuffleServer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1336406342
Thanks @leixm There are some questions.
1. Do you have a POC test? Could you attach test result?
2. Do you compare with other plans, such as Uber, Alibaba RSS, Bytedance RSS. Could we reuse their plan?
3. Do our client need off heap memory to read the data from the server?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1257878986
> Could we use grpc zero-copy to solve this problem?
>
> By the way, Alluxio also use Grpc to transfer data, I think the problem we encountered also exist in Alluxio. We could refer https://dzone.com/articles/moving-from-apache-thrift-to-grpc-a-perspective-fr?utm_medium=tumblr&utm_source=dlvr.it&utm_campaign=Feed%3A%20dzone%2Fintegration
It seems that it use protobuf to serialize the data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1436201523
Meeting link https://meeting.tencent.com/dm/nVx6mxO8vIrc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1407593171
Hi, @leixm xianming, has this issue already been addressed in your internal prod env? Do you any ETA for the PR to be uploaded?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #133: [Umbrella] Netty replace Grpc on data transfer
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1502613193
> I believe we should do it in baby steps, the netty rpc framework should be introduced first, then the off-heap memory management, then unified memory management and spill.
Yes. Let's reserve this
> I haven't thought it throughly, but I believe we can learn from the best.
https://github.com/flink-extended/flink-remote-shuffle is a good example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1424048433
Can I add some ISSUE? Let's get involved. @jerqi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1290046207
@leixm @zuston Do you want to discuss this issue through a meeting? I will start a meeting to discuss the issue #80, I want to discuss this issue, too. There are some other issues which we need to discuss, so I will send a email to our dev mail list, and select a proper date to start the meeting. You can tell me what time you are free by the email.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1425402691
> Good idea, but the current performance bottleneck is mainly in ShuffleServer, we can improve the throughput of ShuffleServer first.
You misunderstand me. When shuffle server read the local file, we use `sendfile` api to improve our shuffle server performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1434249743
okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by GitBox <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1336365287
Currently, we use Uniffle version 0.6.0 internally. When there are tasks running, the GC of Shuffle Server will take up to 40% of the cpu time, and FullGC will appear occasionally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1427944971
> @leixm Have you do some performance tests? @advancedxy @Gustfh @smallzhongfeng @zuston @leixm @kaijchen How about every wednesday 10:15 am every week for this issue regular meeting?
It's ok for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1430656002
> > The performance of disk is limited. Every HDD is 100MB/s. Every SDD is 1GB/s. Usually we test this with 1TB tpcds and 1TB Terasort. If we can get performance improve, we can merge some prs first.
>
> How many ShuffleServer nodes were there in the previous test?
3 - 6 nodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1425100400
After offline discussion, we should involve more people to participate in this issue. @zuston @leixm @advancedxy @kaijchen @smallzhongfeng @xianjingfeng Maybe we should have a regular meeting to discuss this issue continuously. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] Gustfh commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "Gustfh (via GitHub)" <gi...@apache.org>.
Gustfh commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1425135692
recently, we also working on this feature, since we found the problem is on the grpc, later we will share our result about this poc.
+1 for the regular meeting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #133: [Improvement] Netty replace Grpc on data transfer
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #133:
URL: https://github.com/apache/incubator-uniffle/issues/133#issuecomment-1425280933
> > https://github.com/leixm/incubator-uniffle/tree/netty_poc_2
>
> Great. Do you think is it possible to submit an WIP pr now?
>
> > It is still in the development process and has not been applied to the production environment.
>
> No problem, let's create issues/prs targets your branch and discuss more.
Should we submit WIP PR first or do a performance test first to prove that the code is valid, then submit WIP PR and continue to complete the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org