You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by GitBox <gi...@apache.org> on 2021/11/08 13:54:01 UTC

[GitHub] [systemds] ywcb00 opened a new pull request #1446: [SYSTEMDS-3206] Federated Aggregate Binary and MM Instruction

ywcb00 opened a new pull request #1446:
URL: https://github.com/apache/systemds/pull/1446


   Hi,
   This PR fixes two issues with the _AggregateBinaryFEDInstruction_ and the _MMFEDInstruction_.
   1. It removes the seperate alignment check inside the two instructions where there is no verification that the range-aligned federated data partitions are located on the same federated worker. Since the case of having two aligned matrices is already covered by the very first if branch, I removed the whole case containing the local alignment check.
   2. Since the federated spark instructions do not automatically create the output variables yet, we need to create the output matrices by an additional PUT_VAR request to the federated worker. However, this PR adds some of those PUT_VAR requests that were missing inside the _MMFEDInstruction_.
   
   In addition, this PR addresses the problem of creating an endless loop between the _acquireReadIntern()_ and the _readBlobFromHDFS()_ method (mentioned in #1390) by adding the constraint to read the file from federated even if it is broadcast but does not exist on hdfs.
   
   Thanks for review :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [systemds] Baunsgaard closed pull request #1446: [SYSTEMDS-3206] Federated Execution w/ Misaligned Matrices

Posted by GitBox <gi...@apache.org>.
Baunsgaard closed pull request #1446:
URL: https://github.com/apache/systemds/pull/1446


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [systemds] ywcb00 edited a comment on pull request #1446: [SYSTEMDS-3206] Federated Execution w/ Misaligned Matrices

Posted by GitBox <gi...@apache.org>.
ywcb00 edited a comment on pull request #1446:
URL: https://github.com/apache/systemds/pull/1446#issuecomment-970159882


   When I extended the tests with misaligned matrices (FederatedMisAlignedTest), I discovered two more issues with the _BinaryMatrixMatrixFEDInstruction_ and the _AppendFEDInstruction_.
   The BinaryMatrixMatrixFEDInstruction was throwing an Exception when called with two federated matrices which are not aligned. I changed it to not throwing an exception and pulling one of the federated matrices to the driver where we can then handle it like a instruction with one federated matrix.
   I've added support for the case of having two aligned federated matrices to the AppendFEDInstruction.
   
   I think this PR is now ready for review :rocket: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [systemds] ywcb00 commented on pull request #1446: [SYSTEMDS-3206] Federated Execution w/ Misaligned Matrices

Posted by GitBox <gi...@apache.org>.
ywcb00 commented on pull request #1446:
URL: https://github.com/apache/systemds/pull/1446#issuecomment-970159882


   When I extended the tests with misaligned matrices (FederatedMisalignedTest), I discovered two more issues with the _BinaryMatrixMatrixFEDInstruction_ and the _AppendFEDInstruction_.
   The BinaryMatrixMatrixFEDInstruction was throwing an Exception when called with two federated matrices which are not aligned. I changed it to not throwing an exception and pulling one of the federated matrices to the driver where we can then handle it like a instruction with one federated matrix.
   I've added support for the case of having two aligned federated matrices to the AppendFEDInstruction.
   
   I think this PR is now ready for review :rocket: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org