[incubator-pinot] branch master updated: Change the realtime stats file to a new name (#3685)

[ne...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

nehapawar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
     new c42af32  Change the realtime stats file to a new name (#3685)
c42af32 is described below

commit c42af32b2fae92c741a68c59ae90a1c19061887e
Author: Neha Pawar <np...@linkedin.com>
AuthorDate: Fri Jan 11 18:27:34 2019 -0800

    Change the realtime stats file to a new name (#3685)
    
    The stats files generated in realtime servers will no longer be deserializable. This is because when serializing, the older objects were used, the ones with "com.linkedin". After the package name change, the newer objects will be used to deserialize, the ones with "org.apache", and the deserialization will fail.
    Changing the stats file name from "stats.ser" to "segment-stats.ser". We will start populating stats afresh in all realtime tables.
    Please note: stats from stats.ser will not be considered once the new server is started.
---
 .../data/manager/realtime/RealtimeTableDataManager.java |   4 +++-
 .../realtime/impl/RealtimeSegmentStatsHistoryTest.java  |   2 +-
 .../data/realtime-segment-stats-history-v1.ser          | Bin 822 -> 832 bytes
 3 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
index ff8a8b8..d24902b 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
@@ -65,7 +65,9 @@ public class RealtimeTableDataManager extends BaseTableDataManager {
   private RealtimeSegmentStatsHistory _statsHistory;
   private final Semaphore _segmentBuildSemaphore;
 
-  private static final String STATS_FILE_NAME = "stats.ser";
+  // The old name of the stats file used to be stats.ser which we changed when we moved all packages
+  // from com.linkedin to org.apache because of not being able to deserialize the old files using the newer classes
+  private static final String STATS_FILE_NAME = "segment-stats.ser";
   private static final String CONSUMERS_DIR = "consumers";
 
   // Topics tend to have similar cardinality for values across partitions consumed during the same time.
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/realtime/impl/RealtimeSegmentStatsHistoryTest.java b/pinot-core/src/test/java/org/apache/pinot/core/realtime/impl/RealtimeSegmentStatsHistoryTest.java
index a1a3c79..3b4a418 100644
--- a/pinot-core/src/test/java/org/apache/pinot/core/realtime/impl/RealtimeSegmentStatsHistoryTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/realtime/impl/RealtimeSegmentStatsHistoryTest.java
@@ -222,7 +222,7 @@ public class RealtimeSegmentStatsHistoryTest {
   // This test attempts to ensure that future modifications to RealtimeSegmentStatsHistory does not prevent the software
   // from reading data serialized by earlier versions. The serialized data has one segment, with 2 columns -- "v1col1" and
   // "v1col2".
-  //@Test
+  @Test
   public void testVersion1() throws Exception {
     final String fileName = "realtime-segment-stats-history-v1.ser";
     File v1StatsFile = new File(TestUtils.getFileFromResourceUrl(RealtimeSegmentStatsHistoryTest.class.getClassLoader().getResource("data")), fileName);
diff --git a/pinot-core/src/test/resources/data/realtime-segment-stats-history-v1.ser b/pinot-core/src/test/resources/data/realtime-segment-stats-history-v1.ser
index e0f09dc..06519f8 100644
Binary files a/pinot-core/src/test/resources/data/realtime-segment-stats-history-v1.ser and b/pinot-core/src/test/resources/data/realtime-segment-stats-history-v1.ser differ


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org