[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3686: [CARBONDATA-3759]Refactor segmentRefreshInfo and fix cache issue in multiple application scenario

[GitBox <gi...@apache.org>]

ajantha-bhat commented on a change in pull request #3686: [CARBONDATA-3759]Refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#discussion_r405437861
 
 

 ##########
 File path: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
 ##########
 @@ -727,32 +730,54 @@ public TableSegmentRefresher getTableSegmentRefresher(CarbonTable table) {
     private Map<String, Boolean> manualSegmentRefresh = new HashMap<>();
 
     TableSegmentRefresher(CarbonTable table) {
-      SegmentUpdateStatusManager statusManager = new SegmentUpdateStatusManager(table);
-      SegmentUpdateDetails[] updateStatusDetails = statusManager.getUpdateStatusDetails();
-      for (SegmentUpdateDetails updateDetails : updateStatusDetails) {
-        UpdateVO updateVO = statusManager.getInvalidTimestampRange(updateDetails.getSegmentName());
+      SegmentStatusManager segmentStatusManager =
+          new SegmentStatusManager(table.getAbsoluteTableIdentifier());
+      List<Segment> validSegments;
+      try {
+        validSegments = segmentStatusManager.getValidAndInvalidSegments().getValidSegments();
+      } catch (IOException e) {
+        LOGGER.error("Error while getting the valid segments.", e);
+        throw new RuntimeException(e);
+      }
+      for (Segment segment : validSegments) {
+        UpdateVO updateVO =
+            SegmentUpdateStatusManager.getInvalidTimestampRange(segment.getLoadMetadataDetails());
         SegmentRefreshInfo segmentRefreshInfo;
-        if (updateVO != null && updateVO.getLatestUpdateTimestamp() != null) {
-          segmentRefreshInfo = new SegmentRefreshInfo(updateVO.getLatestUpdateTimestamp(), 0);
+        if (updateVO != null && updateVO.getLatestUpdateTimestamp() != null
+            || segment.getSegmentFileName() != null) {
+          long segmentFileTimeStamp = FileFactory.getCarbonFile(CarbonTablePath
+              .getSegmentFilePath(table.getTablePath(), segment.getSegmentFileName()))
+              .getLastModifiedTime();
+          segmentRefreshInfo =
+              new SegmentRefreshInfo(updateVO.getLatestUpdateTimestamp(), 0, segmentFileTimeStamp);
         } else {
-          segmentRefreshInfo = new SegmentRefreshInfo(0L, 0);
+          segmentRefreshInfo = new SegmentRefreshInfo(0L, 0, 0L);
 
 Review comment:
   So, old store (without having segment file). This cache clean up fail ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services