[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #6567: Enable segment upload to a realtime table

[GitBox <gi...@apache.org>]

mcvsubbu commented on a change in pull request #6567:
URL: https://github.com/apache/incubator-pinot/pull/6567#discussion_r573367216



##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
##########
@@ -245,21 +245,28 @@ private SuccessResponse uploadSegment(@Nullable String tableName, FormDataMultiP
         LOGGER.info("Uploading a segment {} to table: {}, push type {}, (Derived from segment metadata)", segmentName, tableName, uploadType);
       }
 
-      String offlineTableName = TableNameBuilder.OFFLINE.tableNameWithType(rawTableName);
+      String tableNameWithType;
+      if (_pinotHelixResourceManager.isRealtimeOnlyTable(rawTableName)) {
+        tableNameWithType = TableNameBuilder.REALTIME.tableNameWithType(rawTableName);
+      } else {
+        tableNameWithType = TableNameBuilder.OFFLINE.tableNameWithType(rawTableName);
+      }
       String clientAddress = InetAddress.getByName(request.getRemoteAddr()).getHostName();
       LOGGER.info("Processing upload request for segment: {} of table: {} from client: {}, ingestion descriptor: {}",
-          segmentName, offlineTableName, clientAddress, ingestionDescriptor);
+          segmentName, tableNameWithType, clientAddress, ingestionDescriptor);
 
-      // Skip segment validation if upload only segment metadata
-      if (uploadType != FileUploadDownloadClient.FileUploadType.METADATA) {
+      // Skip segment validation if upload only segment metadata or it is a realtime table segment.
+      // TODO Perform a validation check for realtime segments too.

Review comment:
       For realtime segments, one of the validations that need to be performed is that the segment is generated off the same stream endpoint as the subscription of realtime table. So, the segment generation should include something in the metadata to indicate this. Otherwise, queries will return bad results.
   
   For example, if the same topic was consumed from one data center, it is not possible to copy that segment over, and "upload" to the realtime table in another data center, even though the stream data, schema, etc. are same. We need some indication of the stream source (so that offsets are aligned).
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org