[GitHub] [systemds] ywcb00 opened a new pull request #1544: [SYSTEMDS-3185] Federated Lineage Transfer for PUT Data

[GitBox <gi...@apache.org>]

ywcb00 opened a new pull request #1544:
URL: https://github.com/apache/systemds/pull/1544


   Hi,
   This PR introduces the mechanics for tranferring the lineage trace of a data object to the federated worker.
   For now, we are including only the lineage trace of matrices which come from the data gen operation (e.g. _rand()_), as they have the respective lineage item set in their _CacheableData_ objects.
   In order to include the lineage trace for all matrices, we need to pass it through the _broadcast_ and _broadcastSliced_ methods to the creation of the actual PUT_VAR FederatedRequest. I decided to introduce the mechanics at first, and change the _broadcast_ methods with the next PR. This will allow for a better discussion about where to include the lineage trace, since then we can decide per instruction whether to put the lineage trace to the federated site or not.
   However, I first tried to set the _LineageItem_ into the _CacheableData_ object at every instruction, but this leads to an increase in the size of the _CacheableData_ object. Therefore, I just removed it again and changed the tests such that they have only one _LineageItem_ (from _DataGen_) in the lineage trace of the _PUT_ object for now.
   
   Thanks for review :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [systemds] phaniarnab closed pull request #1544: [SYSTEMDS-3185] Federated Lineage Transfer for PUT Data

[GitBox <gi...@apache.org>]

phaniarnab closed pull request #1544:
URL: https://github.com/apache/systemds/pull/1544


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [systemds] phaniarnab commented on pull request #1544: [SYSTEMDS-3185] Federated Lineage Transfer for PUT Data

[GitBox <gi...@apache.org>]

phaniarnab commented on pull request #1544:
URL: https://github.com/apache/systemds/pull/1544#issuecomment-1059411255


   Thanks for the initial lineage trace transfer logic @ywcb00. 
   The code changes look good to me. I am only a bit skeptical about replacing the `HashMap` of our live variable map with a `ConcurrentHashMap`. However, we can discuss that later. I merged the changes with minor fixes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org