You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by GitBox <gi...@apache.org> on 2022/01/22 17:28:14 UTC

[GitHub] [systemds] ywcb00 opened a new pull request #1521: [SYSTEMDS-3185] Federated Read Cache

ywcb00 opened a new pull request #1521:
URL: https://github.com/apache/systemds/pull/1521


   Hi,
   this PR introduces the _FederatedReadCache_ for the federated worker. The _FederatedReadCache_ is basically a map that maps from filename to a data object. In order to allow and optimize for concurrent READ_VAR requests on the federated site, the data object is wrapped inside a placeholder (_ReadCacheEntry_) so that only one thread reads the object from fs while the other threads wait for the data to be placed inside the placeholder.
   In addition, this PR enables multithreaded processing of requests at the federated worker with a maximal threadpool size of 4 times the number of cpu processors.
   
   Thanks for review :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [systemds] mboehm7 commented on pull request #1521: [SYSTEMDS-3185] Federated Read Cache

Posted by GitBox <gi...@apache.org>.
mboehm7 commented on pull request #1521:
URL: https://github.com/apache/systemds/pull/1521#issuecomment-1019549855


   LGTM - during the merge I fixed the following issues (which were only partially related to the change):
   * On the multi-tenant tests, I got a `CreateProcess error=206, The filename or extension is too long` and hence modified these tests to pass the file name just once and then concatenate within dml
   * Fixed the javadoc issue (missing read cache parameter)
   * Fixed federated unary and binary aggregates (forced federated output was not waiting to completion; with the multi-threaded event loop the sequence of operations was no longer guaranteed)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [systemds] asfgit closed pull request #1521: [SYSTEMDS-3185] Federated Read Cache

Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #1521:
URL: https://github.com/apache/systemds/pull/1521


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org