You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by GitBox <gi...@apache.org> on 2020/11/15 19:26:06 UTC
[GitHub] [systemds] Baunsgaard opened a new pull request #1106: [MINOR] Federated Type Fixes
Baunsgaard opened a new pull request #1106:
URL: https://github.com/apache/systemds/pull/1106
I hope these changes don't overlap to much with yours @mboehm7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard merged pull request #1106: [MINOR] Federated Type Fixes
Posted by GitBox <gi...@apache.org>.
Baunsgaard merged pull request #1106:
URL: https://github.com/apache/systemds/pull/1106
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] mboehm7 commented on pull request #1106: [MINOR] Federated Type Fixes
Posted by GitBox <gi...@apache.org>.
mboehm7 commented on pull request #1106:
URL: https://github.com/apache/systemds/pull/1106#issuecomment-727628915
no, seems fine and I'm also done for now. In order to run the test pipeline fully federated, it would be good though to extend the removeEmpty support for selection vectors, which makes the splitting vastly faster and preserves a nice balance across federated partitions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1106: [MINOR] Federated Type Fixes
Posted by GitBox <gi...@apache.org>.
Baunsgaard commented on pull request #1106:
URL: https://github.com/apache/systemds/pull/1106#issuecomment-727640449
> A general question on this: isn't a full block just a row or column partitioned federated matrix and hence the existing operations can directly be applied. Yes, we could implement them more efficiently (basic forwarding, no aggregation, etc) but as this is a special case I'm not sure it's worth the implementation complexity. Any thoughts?
yes, but the basic issue is that when you only have one federated location, should i initialize as a row or a col?
Currently this dilemma results in scenarios where the data is aggregated from the federated sites when we have one worker.
I don't intend to optimize particularly for it, but with this addition we are able to leverage both our row and col optimized operations in this special case with not to much extra code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] mboehm7 commented on pull request #1106: [MINOR] Federated Type Fixes
Posted by GitBox <gi...@apache.org>.
mboehm7 commented on pull request #1106:
URL: https://github.com/apache/systemds/pull/1106#issuecomment-727640839
ok, increasing the generality of implementations is a good idea. In that case I would recommend to not probe for type FULL in individual operations, but put methods isRowPartitioned (ROW or FULL), isColumnPartitions (COL or FULL) into the enum type and then probe these methods from the instructions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] mboehm7 commented on pull request #1106: [MINOR] Federated Type Fixes
Posted by GitBox <gi...@apache.org>.
mboehm7 commented on pull request #1106:
URL: https://github.com/apache/systemds/pull/1106#issuecomment-727639604
A general question on this: isn't a full block just a row or column partitioned federated matrix and hence the existing operations can directly be applied. Yes, we could implement them more efficiently (basic forwarding, no aggregation, etc) but as this is a special case I'm not sure it's worth the implementation complexity. Any thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Baunsgaard commented on pull request #1106: [MINOR] Federated Type Fixes
Posted by GitBox <gi...@apache.org>.
Baunsgaard commented on pull request #1106:
URL: https://github.com/apache/systemds/pull/1106#issuecomment-727640947
> ok, increasing the generality of implementations is a good idea. In that case I would recommend to not probe for type FULL in individual operations, but put methods isRowPartitioned (ROW or FULL), isColumnPartitions (COL or FULL) into the enum type and then probe these methods from the instructions.
Ah good idea, I forgot java can do this with Enums
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org