You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by GitBox <gi...@apache.org> on 2022/11/25 13:45:14 UTC
[GitHub] [systemds] phaniarnab opened a new pull request, #1736: [SYSTEMDS-3469] New operator ordering to maximize inter-op parallelism
phaniarnab opened a new pull request, #1736:
URL: https://github.com/apache/systemds/pull/1736
This patch introduces a new heuristic-based operator linearization order, which aims to maximize inter-operator parallelism among Spark and local operators. We first traverse the LOP DAGs to collect the roots of the Spark operator chains and the number of Spark instructions in all subDAGs. We then first place the Spark operator chains followed by the CP lanes. Finally, we place the appropriate asynchronous operators to trigger the Spark operator chains in parallel.
This change along with the future-based execution of Spark actions and a manual reuse of partitioned broadcast variables improve lmDS by 2x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] phaniarnab closed pull request #1736: [SYSTEMDS-3469] New operator ordering to maximize inter-op parallelism
Posted by GitBox <gi...@apache.org>.
phaniarnab closed pull request #1736: [SYSTEMDS-3469] New operator ordering to maximize inter-op parallelism
URL: https://github.com/apache/systemds/pull/1736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] BACtaki commented on pull request #1736: [SYSTEMDS-3469] New operator ordering to maximize inter-op parallelism
Posted by GitBox <gi...@apache.org>.
BACtaki commented on PR #1736:
URL: https://github.com/apache/systemds/pull/1736#issuecomment-1327727033
> This change along with the future-based execution of Spark actions and a manual reuse of partitioned broadcast variables improve lmDS by 2x.
@phaniarnab Just curious: what is lmDS?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] phaniarnab commented on pull request #1736: [SYSTEMDS-3469] New operator ordering to maximize inter-op parallelism
Posted by GitBox <gi...@apache.org>.
phaniarnab commented on PR #1736:
URL: https://github.com/apache/systemds/pull/1736#issuecomment-1327789835
A builtin for linear regression direct solve :P @BACtaki
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] BACtaki commented on pull request #1736: [SYSTEMDS-3469] New operator ordering to maximize inter-op parallelism
Posted by GitBox <gi...@apache.org>.
BACtaki commented on PR #1736:
URL: https://github.com/apache/systemds/pull/1736#issuecomment-1327861982
> A builtin for linear regression direct solve :P @BACtaki
Thanks for clarifying, now I know :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org