[GitHub] [pinot] jeffreyliu34 commented on a change in pull request #8273: Add prefixesToDropFromFields config for renaming fields upon ingestion

[GitBox <gi...@apache.org>]

jeffreyliu34 commented on a change in pull request #8273:
URL: https://github.com/apache/pinot/pull/8273#discussion_r819895640



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/ComplexTypeTransformer.java
##########
@@ -279,6 +294,23 @@ protected void flattenMap(GenericRow record, List<String> columns) {
     }
   }
 
+  /**
+   *
+   */
+  @VisibleForTesting
+  protected void dropPrefixes(GenericRow record) {
+    List<String> columns = new ArrayList<>(record.getFieldToValueMap().keySet());

Review comment:
       Good point. Since this was mainly intended for unnested fields, I was thinking it to be a rare edge case for prefixes matching regular columns since they'd lack the delimiter (ie "after." vs "after"), and thought it might help to have more flexibility to drop prefixes from regular columns if desired. I understand the concern though, so it can be updated it to drop prefixes only from columns generated by the ComplexTypeTransformer. 
   
   Initially, I thought about doing this when generating the `flattenName`, but this would affect the `unnestCollection` logic here - https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/ComplexTypeTransformer.java#L154-L157 since we're flattening the map first (where we would have dropped prefixes) before unnesting any collections. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org