You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by GitBox <gi...@apache.org> on 2022/02/05 11:42:50 UTC
[GitHub] [systemds] ywcb00 opened a new pull request #1535: [SYSTEMDS-3185] Worker-specific Multithreading of Operations
ywcb00 opened a new pull request #1535:
URL: https://github.com/apache/systemds/pull/1535
Hi,
This PR adds the _MultiThreadedOperator_ class as a superclass for all operators supporting multithreading. This allows us to replace the number of threads of the operator with a worker-specific amount after parsing the received instruction from the EXEC_INST request.
With this PR, we are setting the number of threads to the amount of processors available to the JVM of the federated worker whenever the operator allows for multithreading (i.e. a subclass of _MultiThreadedOperator_).
Thanks for review :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] asfgit closed pull request #1535: [SYSTEMDS-3185] Worker-specific Multithreading of Operations
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #1535:
URL: https://github.com/apache/systemds/pull/1535
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1535: [SYSTEMDS-3185] Worker-specific Multithreading of Operations
Posted by GitBox <gi...@apache.org>.
Baunsgaard commented on pull request #1535:
URL: https://github.com/apache/systemds/pull/1535#issuecomment-1031344094
@mboehm7
Python test failed because when reading in a file for some reason we to import Apache commons-compress and that is not available in our bin release.
The only place where we have automated the testing of our bin release is in Python. We should add a test of our normal system that use our bin release.
See 336ac8b4a0c698094d6f3300cb79d08e5ccff06e for fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] mboehm7 commented on pull request #1535: [SYSTEMDS-3185] Worker-specific Multithreading of Operations
Posted by GitBox <gi...@apache.org>.
mboehm7 commented on pull request #1535:
URL: https://github.com/apache/systemds/pull/1535#issuecomment-1030922713
LGTM - thanks for the patch @ywcb00. This is a nice, generic way of handling degree of parallelism at the federated workers. Before merging it in, I'd like to understand why the Python tests are now failing in this and other PRs but not on main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] mboehm7 commented on pull request #1535: [SYSTEMDS-3185] Worker-specific Multithreading of Operations
Posted by GitBox <gi...@apache.org>.
mboehm7 commented on pull request #1535:
URL: https://github.com/apache/systemds/pull/1535#issuecomment-1031346769
ok thanks for the clarification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] mboehm7 commented on pull request #1535: [SYSTEMDS-3185] Worker-specific Multithreading of Operations
Posted by GitBox <gi...@apache.org>.
mboehm7 commented on pull request #1535:
URL: https://github.com/apache/systemds/pull/1535#issuecomment-1058439828
LGTM again, and sorry for the delay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org