[GitHub] [systemds] sebwrede commented on pull request #1199: [SYSTEMDS-2604] Federated compile and federated output propagation

[GitBox <gi...@apache.org>]

sebwrede commented on pull request #1199:
URL: https://github.com/apache/systemds/pull/1199#issuecomment-820219888


   > For my understanding, did you mean that this flag will help to execute a chain of federated operations before fetching the outputs to the coordinator? Is it not the case today?
   
   This PR adds a flag to compile the federated instructions during compile-time instead of simply converting the CP instructions to FED instructions during runtime. Later, the decision to compile to FED will be expanded so that it can base the decision on several scenarios, including the scenario where a chain of federated operations can be executed without sharing the data with the coordinator. Many of the FED instructions are currently implemented in a way that keeps the output federated, but this decision of keeping the output federated needs to be dependent on the flag so that we can have some kind of "FED planner" which compiles this information. 
   At the moment, the federated instructions do not take the privacy constraints into account. When we have a FED planner, it could discover a privacy constraint on some input data and then compile the instructions accordingly. This may in some cases mean that it does not just blindly pick the chain of federated instructions, but instead considers the privacy constraint and makes a new plan that adheres to the constraint. This is just an example of how the FED planner will be convenient later on. 
   I implemented this as a federated output flag to begin with, but this will likely be expanded later on since there are more decisions than just federated vs. local. I will deal with this in later PRs when I have a basic FED planner and support federated compilation of more operations. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org