You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/01/27 02:08:06 UTC

[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6494: Support generating derived column during segment load

Jackie-Jiang commented on a change in pull request #6494:
URL: https://github.com/apache/incubator-pinot/pull/6494#discussion_r564973731



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java
##########
@@ -324,8 +345,69 @@ protected void removeColumnV1Indices(String column)
    */
   protected void createColumnV1Indices(String column)
       throws Exception {
+    TableConfig tableConfig = _indexLoadingConfig.getTableConfig();
+    if (tableConfig != null && tableConfig.getIngestionConfig() != null
+        && tableConfig.getIngestionConfig().getTransformConfigs() != null) {
+      List<TransformConfig> transformConfigs = tableConfig.getIngestionConfig().getTransformConfigs();
+      for (TransformConfig transformConfig : transformConfigs) {
+        if (transformConfig.getColumnName().equals(column)) {
+          String transformFunction = transformConfig.getTransformFunction();
+          FunctionEvaluator functionEvaluator = FunctionEvaluatorFactory.getExpressionEvaluator(transformFunction);
+
+          // Check if all arguments exist in the segment
+          // TODO: Support chained derived column
+          boolean skipCreatingDerivedColumn = false;
+          List<String> arguments = functionEvaluator.getArguments();
+          List<ColumnMetadata> argumentsMetadata = new ArrayList<>(arguments.size());
+          for (String argument : arguments) {
+            ColumnMetadata columnMetadata = _segmentMetadata.getColumnMetadataFor(argument);
+            if (columnMetadata == null) {
+              LOGGER.warn(
+                  "Skip creating derived column: {} because argument: {} does not exist in the segment, creating default value column instead",
+                  column, argument);
+              skipCreatingDerivedColumn = true;
+              break;
+            }
+            argumentsMetadata.add(columnMetadata);
+          }
+          if (skipCreatingDerivedColumn) {
+            break;
+          }
+
+          // TODO: Support raw derived column
+          if (_indexLoadingConfig.getNoDictionaryColumns().contains(column)) {
+            LOGGER.warn("Skip creating raw derived column: {}, creating default value column instead", column);
+            break;
+          }
+
+          // TODO: Support multi-value derived column
+          if (!_schema.getFieldSpecFor(column).isSingleValueField()) {
+            LOGGER.warn("Skip creating MV derived column: {}, creating default value column instead", column);
+            break;
+          }
+
+          try {
+            createDerivedColumnV1Indices(column, functionEvaluator, argumentsMetadata);
+            return;
+          } catch (Exception e) {
+            LOGGER.error(

Review comment:
       Hmm, good question. Currently Pinot does not support removing/updating existing index in v3 format, so the only way to re-generate the column is to download a new segment from the controller. Once we have index modification supported, we can add that




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org