You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by GitBox <gi...@apache.org> on 2022/05/01 12:16:56 UTC
[GitHub] [systemds] ywcb00 opened a new pull request, #1601: [SYSTEMDS-3363] Netty Encoding - Buffer Allocation Size
ywcb00 opened a new pull request, #1601:
URL: https://github.com/apache/systemds/pull/1601
Hi,
this PR replaces the ObjectEncoders for encoding federated communication messages (i.e., FederatedRequest and FederatedResponse) with a respective FederatedRequestEncoder and FederatedResponseEncoder. These two new classes extent from ObjectEncoder and override the allocateBuffer method in order to allocate the buffer with an initial capacity corresponding to the FederatedRequest/FederatedResponse instead of the default value of 256.
I've tried estimating the serialized size of the requests and responses to allocate the exact size needed, but haven't found a good solution for it because of the many cases to consider. Hence, the initial capacity in this PR only results from taking into consideration a static offset (FederatedRequest: 512, FederatedResponse: 312), the size of the CacheBlock parameters, and the length of the lineage trace (only for FederatedRequests).
In my experiments with a matrix of dimensions 600.000x200, it reduces the encoding time from 50 seconds to 1.9 seconds. :tada:
Thanks for review :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1601: [SYSTEMDS-3363] Netty Encoding - Buffer Allocation Size
Posted by GitBox <gi...@apache.org>.
Baunsgaard commented on PR #1601:
URL: https://github.com/apache/systemds/pull/1601#issuecomment-1114283531
Nice!
I will take a look tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard closed pull request #1601: [SYSTEMDS-3363] Netty Encoding - Buffer Allocation Size
Posted by GitBox <gi...@apache.org>.
Baunsgaard closed pull request #1601: [SYSTEMDS-3363] Netty Encoding - Buffer Allocation Size
URL: https://github.com/apache/systemds/pull/1601
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1601: [SYSTEMDS-3363] Netty Encoding - Buffer Allocation Size
Posted by GitBox <gi...@apache.org>.
Baunsgaard commented on PR #1601:
URL: https://github.com/apache/systemds/pull/1601#issuecomment-1115019275
I like the change, I am a bit worried for the static offsets but they do not currently seem to fail in any of our cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1601: [SYSTEMDS-3363] Netty Encoding - Buffer Allocation Size
Posted by GitBox <gi...@apache.org>.
Baunsgaard commented on PR #1601:
URL: https://github.com/apache/systemds/pull/1601#issuecomment-1115018529
Merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org