[GitHub] [systemds] phaniarnab commented on a change in pull request #1522: [SYSTEMDS-3185] Lineage-based Reuse of Federated Reads

[GitBox <gi...@apache.org>]

phaniarnab commented on a change in pull request #1522:
URL: https://github.com/apache/systemds/pull/1522#discussion_r800668490



##########
File path: src/main/java/org/apache/sysds/runtime/controlprogram/federated/FederatedWorkerHandler.java
##########
@@ -246,74 +246,43 @@ private FederatedResponse readData(String filename, DataType dataType,
 			// early throwing of exception to avoid infinitely waiting threads for data
 			throw new FederatedWorkerHandlerException("Could not recognize datatype");
 
-		CacheableData<?> cd = _frc.get(filename);
-		if(cd == null) {
+		ExecutionContext ec = ecm.get(tid);
+		LineageItem linItem = new LineageItem(filename);
+
+		CacheableData<?> cd = null;
+
+		if(!LineageCache.reuseRead(Long.toString(id), dataType, linItem, ec)) {
+			if(ReuseCacheType.isNone() || dataType != DataType.MATRIX)
+				// using the read cache if lineage reuse is disabled or the data is not a matrix
+				cd = _frc.get(filename);

Review comment:
       Correct me if I am wrong. It is possible that lineage-based reuse is enabled after the worker is initiated. In that case, the read cache may not be empty. Hence, we may want to lookup both the caches. However, the condition above `_frc.get(filename)` restricts readcache lookup if lineage reuse is enabled.
   I tried to remove the condition but that led to a test failure probably due to an uninitialized readcache.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org