[GitHub] npawar commented on issue #3685: Change the realtime stats file to a new name

[GitBox <gi...@apache.org>]

npawar commented on issue #3685: Change the realtime stats file to a new name
URL: https://github.com/apache/incubator-pinot/pull/3685#issuecomment-454236916
 
 
   If resetting stats is not desirable, the old stats will have to be copied over to the new file. 
   I wrote a tool to help with this translation. This tool was a java project which had both versions of the file `RealtimeSegmentStatsHistory.java` in the same project (`com.linkedin.pinot.core.realtime.impl.RealtimeSegmentStatsHistory` and  `org.apache.pinot.core.realtime.impl.RealtimeSegmentStatsHistory`). It also had a main method which created a new stats file and copied the old stats to this file.
   
   Here's snippets from a tool I wrote.
   
   Main.java
   ```
   import com.linkedin.pinot.core.realtime.impl.RealtimeSegmentStatsHistory;
   import java.io.File;
   import java.io.IOException;
   import java.nio.file.Files;
   
   
   public class Main {
   
       public static void main(String[] args) throws Exception {
   
           String oldStatsFilePath = args[0];
           RealtimeSegmentStatsHistory
               oldHistory = RealtimeSegmentStatsHistory.deserialzeFrom(new File(oldStatsFilePath));
   
           String newStatsFilePath = args[1];
           File newStatsFile = new File(newStatsFilePath);
           Files.deleteIfExists(newStatsFile.toPath());
   
           org.apache.pinot.core.realtime.impl.RealtimeSegmentStatsHistory newHistory;
           try {
               newHistory = org.apache.pinot.core.realtime.impl.RealtimeSegmentStatsHistory.deserialzeFrom(newStatsFile);
           } catch (IOException | ClassNotFoundException e) {
               throw new RuntimeException(
                   "Exception when deserializing stats history from stats file " + newStatsFile.getAbsolutePath(), e);
           }
   
           // copy entries and cursor
           newHistory.setEntries(oldHistory.getEntries());
           newHistory.setCursor(oldHistory.getCursor());
   
           System.out.println(oldHistory.toString());
           System.out.println(newHistory.toString());
   
       }
   }
   ```
   
   Added these methods in the `org.apache.pinot.core.realtime.impl.RealtimeSegmentStatsHistory` file:
   
   setEntries
   ```
   public void setEntries(com.linkedin.pinot.core.realtime.impl.RealtimeSegmentStatsHistory.SegmentStats[] linkedinEntries) {
       for (com.linkedin.pinot.core.realtime.impl.RealtimeSegmentStatsHistory.SegmentStats linkedinEntry : linkedinEntries) {
         if (linkedinEntry == null) {
           continue;
         }
         SegmentStats newSegmentStats = convertFromLinkedinToApache(linkedinEntry);
         addSegmentStats(newSegmentStats);
       }
     }
   ```
   
   
   convertFromLinkedinToApache for SegmentStats:
   ```
   public SegmentStats convertFromLinkedinToApache(
         com.linkedin.pinot.core.realtime.impl.RealtimeSegmentStatsHistory.SegmentStats linkedinSegmentStats) {
       SegmentStats apacheSegmentStats = new SegmentStats();
       apacheSegmentStats.setNumRowsConsumed(linkedinSegmentStats.getNumRowsConsumed());
       apacheSegmentStats.setNumRowsIndexed(linkedinSegmentStats.getNumRowsIndexed());
       apacheSegmentStats.setNumSeconds(linkedinSegmentStats.getNumSeconds());
       apacheSegmentStats.setMemUsedBytes(linkedinSegmentStats.getMemUsedBytes());
       Map<String, com.linkedin.pinot.core.realtime.impl.RealtimeSegmentStatsHistory.ColumnStats> colNameToStats =
           linkedinSegmentStats.getColNameToStats();
       for (Map.Entry<String, com.linkedin.pinot.core.realtime.impl.RealtimeSegmentStatsHistory.ColumnStats> entry : colNameToStats.entrySet()) {
         ColumnStats newColumnStats = convertFromLinkedinToApache(entry.getValue());
         String column = entry.getKey();
         apacheSegmentStats.setColumnStats(column, newColumnStats);
       }
       return apacheSegmentStats;
     }
   ```
   
   
   convertFromLinkedinToApache for ColumnStats:
   ```
   public ColumnStats convertFromLinkedinToApache(
         com.linkedin.pinot.core.realtime.impl.RealtimeSegmentStatsHistory.ColumnStats linkedinColumnStats) {
       ColumnStats apacheColumnStats = new ColumnStats();
       apacheColumnStats.setAvgColumnSize(linkedinColumnStats.getAvgColumnSize());
       apacheColumnStats.setCardinality(linkedinColumnStats.getCardinality());
       return apacheColumnStats;
     }
   ```
   
   Some getters and setters will need to be added to both classes (cursor, entries, segmentStats map etc.)
   
   Ran this tool to convert all `stats.ser` files on realtime servers to `segment-stats.ser`
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org
For additional commands, e-mail: dev-help@pinot.apache.org