[GitHub] [pinot] richardstartin commented on a change in pull request #7405: Introduce Native Text Indices (Core Functionality)

[GitBox <gi...@apache.org>]

richardstartin commented on a change in pull request #7405:
URL: https://github.com/apache/pinot/pull/7405#discussion_r710101805



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/ConstantArcSizeFST.java
##########
@@ -0,0 +1,159 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.utils.nativefst;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.segment.local.utils.nativefst.builders.FSTBuilder;
+
+
+/**
+ * A FST with constant-size arc representation produced directly by
+ * {@link FSTBuilder}.
+ *
+ * @see FSTBuilder
+ */
+public final class ConstantArcSizeFST extends FST {
+  /** Size of the target address field (constant for the builder). */
+  public final static int TARGET_ADDRESS_SIZE = 4;
+
+  /** Size of the flags field (constant for the builder). */
+  public final static int FLAGS_SIZE = 1;
+
+  /** Size of the label field (constant for the builder). */
+  public final static int LABEL_SIZE = 1;
+
+  /**
+   * Size of a single arc structure.
+   */
+  public final static int ARC_SIZE = FLAGS_SIZE + LABEL_SIZE + TARGET_ADDRESS_SIZE;
+
+  /** Offset of the flags field inside an arc. */
+  public final static int FLAGS_OFFSET = 0;
+
+  /** Offset of the label field inside an arc. */
+  public final static int LABEL_OFFSET = FLAGS_SIZE;
+
+  /** Offset of the address field inside an arc. */
+  public final static int ADDRESS_OFFSET = LABEL_OFFSET + LABEL_SIZE;
+  /**
+   * An arc flag indicating the target node of an arc corresponds to a final
+   * state.
+   */
+  public final static int BIT_ARC_FINAL = 1 << 1;
+  /** An arc flag indicating the arc is last within its state. */
+  public final static int BIT_ARC_LAST = 1 << 0;
+  /** A dummy address of the terminal state. */
+  public final static int TERMINAL_STATE = 0;
+  /**
+   * An epsilon state. The first and only arc of this state points either to the
+   * root or to the terminal state, indicating an empty automaton.
+   */
+  private final int _epsilon;
+
+  /**
+   * FST data, serialized as a byte array.
+   */
+  private final byte[] _data;
+
+  private Map<Integer, Integer> _outputSymbols;
+
+  /**
+   * @param data
+   *          FST data. There must be no trailing bytes after the last state.
+   */
+  public ConstantArcSizeFST(byte[] data, int epsilon, Map<Integer, Integer> outputSymbols) {
+    assert epsilon == 0 : "Epsilon is not zero?";
+
+    this._epsilon = epsilon;
+    this._data = data;
+    this._outputSymbols = outputSymbols;
+  }
+
+  @Override
+  public int getRootNode() {
+    return getEndNode(getFirstArc(_epsilon));
+  }
+
+  @Override
+  public int getFirstArc(int node) {
+    return node;
+  }
+
+  @Override
+  public int getArc(int node, byte label) {
+    for (int arc = getFirstArc(node); arc != 0; arc = getNextArc(arc)) {
+      if (getArcLabel(arc) == label) {
+        return arc;
+      }
+    }
+    return 0;
+  }
+
+  @Override
+  public int getNextArc(int arc) {
+    if (isArcLast(arc)) {
+      return 0;
+    }
+    return arc + ARC_SIZE;
+  }
+
+  @Override
+  public byte getArcLabel(int arc) {
+    return _data[arc + LABEL_OFFSET];
+  }
+
+  @Override
+  public int getOutputSymbol(int arc) {
+    return _outputSymbols.get(arc);

Review comment:
       NPE here when `arc` is not in the map.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org