[GitHub] [carbondata] marchpure commented on a change in pull request #3880: [CARBONDATA-3879] Filtering Segmets Optimazation

[GitBox <gi...@apache.org>]

marchpure commented on a change in pull request #3880:
URL: https://github.com/apache/carbondata/pull/3880#discussion_r466265360



##########
File path: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java
##########
@@ -389,14 +389,16 @@ protected FileSplit makeSplit(String segmentId, String filePath, long start, lon
 
   public void updateLoadMetaDataDetailsToSegments(List<Segment> validSegments,
       List<org.apache.carbondata.hadoop.CarbonInputSplit> prunedSplits) {
+    Map<String, Segment> validSegmentsMap = validSegments.stream()

Review comment:
       Appreciate for that good suggestion. But here we can only use Map instead of Set. 
   
   Reason: The The pseudo-code for this function is shown as below.
   
           // **0. segments.hashcode is segmentno, to when we compare 2 segments, only segmentno will be compared**
           if (validSegments.contains(segmentInSplit)) {
   		  **1. fetch the segment from validSegments** 
   		  Segment segmentInValidSegment <- fetch the segment from validSegments
   		  2. 
             segmentInSplit.setLoadMetadataDetails(segmentInValidSegment.getLoadMetadataDetails);
           }
   
   For SET. it's hard to fetch the segment from validSegments.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org