You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/02/10 19:27:24 UTC

[GitHub] [incubator-pinot] siddharthteotia opened a new pull request #5061: Improvements to data anonymizer tool

siddharthteotia opened a new pull request #5061: Improvements to data anonymizer tool
URL: https://github.com/apache/incubator-pinot/pull/5061
 
 
   Following improvements are made based on the the latest usage to enhance an internal test framework
   
   - The global dictionary code was taking too long since the initial implementation was array based and used linear search while building the global dictionary. We had earlier avoided use of map/set to minimize the heap overhead as opposed to optimizing for performance. But it turns out that for higher cardinalities, linear search is significantly slowing down this tool. This PR implements a Map based global dictionary to keep the bound to O(logN).
   
   - Fixed bug for multi-value support. Avro API needs Object[] to be casted as Arrays.asList
   
   - Users of this tool might use a partial production dataset (as we are planning to do so) to workaround the memory requirements of this tool. In this case, the global dictionary will not have each and every value from the dictionary of original segment. So the query generator needs to be aware of this and should ignore such queries for which it is not able to rewrite the predicate since the original value never made it to the global dictionary as it was not a part of the chosen subset of segments.
   
   A new global dictionary interface is implemented. Existing array based implementation is kept intact and a new map based concrete implementation is written.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] jackjlli merged pull request #5061: Improvements to data anonymizer tool

Posted by GitBox <gi...@apache.org>.
jackjlli merged pull request #5061: Improvements to data anonymizer tool
URL: https://github.com/apache/incubator-pinot/pull/5061
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] siddharthteotia commented on issue #5061: Improvements to data anonymizer tool

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #5061: Improvements to data anonymizer tool
URL: https://github.com/apache/incubator-pinot/pull/5061#issuecomment-584307927
 
 
   cc @mcvsubbu , @jackjlli 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-io commented on issue #5061: Improvements to data anonymizer tool

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #5061: Improvements to data anonymizer tool
URL: https://github.com/apache/incubator-pinot/pull/5061#issuecomment-584332757
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5061?src=pr&el=h1) Report
   > Merging [#5061](https://codecov.io/gh/apache/incubator-pinot/pull/5061?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/f5fe4af026824c2096559c62dfaac3ef52542b79?src=pr&el=desc) will **decrease** coverage by `0.2%`.
   > The diff coverage is `0%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/5061/graphs/tree.svg?width=650&token=4ibza2ugkz&height=150&src=pr)](https://codecov.io/gh/apache/incubator-pinot/pull/5061?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #5061      +/-   ##
   ==========================================
   - Coverage   39.63%   39.42%   -0.21%     
   ==========================================
     Files        1182     1185       +3     
     Lines       63135    63394     +259     
     Branches     9287     9315      +28     
   ==========================================
   - Hits        25022    24992      -30     
   - Misses      35935    36239     +304     
   + Partials     2178     2163      -15
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5061?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...inot/tools/admin/command/AnonymizeDataCommand.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtdG9vbHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3Rvb2xzL2FkbWluL2NvbW1hbmQvQW5vbnltaXplRGF0YUNvbW1hbmQuamF2YQ==) | `0% <ø> (ø)` | :arrow_up: |
   | [...t/tools/anonymizer/MapBasedGlobalDictionaries.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtdG9vbHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3Rvb2xzL2Fub255bWl6ZXIvTWFwQmFzZWRHbG9iYWxEaWN0aW9uYXJpZXMuamF2YQ==) | `0% <0%> (ø)` | |
   | [...tools/anonymizer/ArrayBasedGlobalDictionaries.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtdG9vbHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3Rvb2xzL2Fub255bWl6ZXIvQXJyYXlCYXNlZEdsb2JhbERpY3Rpb25hcmllcy5qYXZh) | `0% <0%> (ø)` | |
   | [.../tools/anonymizer/PinotDataAndQueryAnonymizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtdG9vbHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3Rvb2xzL2Fub255bWl6ZXIvUGlub3REYXRhQW5kUXVlcnlBbm9ueW1pemVyLmphdmE=) | `0% <0%> (ø)` | |
   | [...ller/validation/OfflineSegmentIntervalChecker.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL09mZmxpbmVTZWdtZW50SW50ZXJ2YWxDaGVja2VyLmphdmE=) | `12.82% <0%> (-67.95%)` | :arrow_down: |
   | [...not/broker/requesthandler/RangeMergeOptimizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvUmFuZ2VNZXJnZU9wdGltaXplci5qYXZh) | `33.82% <0%> (-36.77%)` | :arrow_down: |
   | [.../org/apache/pinot/client/PinotClientException.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1Bpbm90Q2xpZW50RXhjZXB0aW9uLmphdmE=) | `33.33% <0%> (-33.34%)` | :arrow_down: |
   | [...apache/pinot/common/metrics/ValidationMetrics.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9WYWxpZGF0aW9uTWV0cmljcy5qYXZh) | `47.45% <0%> (-28.82%)` | :arrow_down: |
   | [...inot/client/JsonAsyncHttpPinotClientTransport.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0pzb25Bc3luY0h0dHBQaW5vdENsaWVudFRyYW5zcG9ydC5qYXZh) | `62% <0%> (-14%)` | :arrow_down: |
   | [...r/dociditerators/RangelessBitmapDocIdIterator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9SYW5nZWxlc3NCaXRtYXBEb2NJZEl0ZXJhdG9yLmphdmE=) | `66.66% <0%> (-11.12%)` | :arrow_down: |
   | ... and [57 more](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5061?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5061?src=pr&el=footer). Last update [f5fe4af...7f60204](https://codecov.io/gh/apache/incubator-pinot/pull/5061?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5061: Improvements to data anonymizer tool

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #5061: Improvements to data anonymizer tool
URL: https://github.com/apache/incubator-pinot/pull/5061#discussion_r377274727
 
 

 ##########
 File path: pinot-tools/src/main/java/org/apache/pinot/tools/anonymizer/MapBasedGlobalDictionaries.java
 ##########
 @@ -0,0 +1,278 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.tools.anonymizer;
+
+import java.io.BufferedWriter;
+import java.io.FileWriter;
+import java.io.PrintWriter;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.TreeMap;
+import org.apache.commons.lang.RandomStringUtils;
+import org.apache.pinot.core.segment.index.ColumnMetadata;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.ByteArray;
+
+
+public class MapBasedGlobalDictionaries implements GlobalDictionaries {
+  private final static int INT_BASE_VALUE = 1000;
+  private final static long LONG_BASE_VALUE = 100000;
+  private static final float FLOAT_BASE_VALUE = 100.23f;
+  private static final double DOUBLE_BASE_VALUE = 1000.2375;
+
+  private final Map<String, OrigAndDerivedValueHolder> _columnToGlobalDictionary;
+
+  MapBasedGlobalDictionaries() {
+    _columnToGlobalDictionary = new HashMap<>();
+  }
+
+  /**
+   * First step towards building global dictionary
+   * by inserting the original values from segments
+   * into global dictionary
+   * @param origValue original value
+   * @param column column name
+   * @param columnMetadata column metadata
+   * @param cardinality total cardinality of column
+   */
+  @Override
+  public void addOrigValueToGlobalDictionary(
+      Object origValue,
+      String column,
+      ColumnMetadata columnMetadata,
+      int cardinality) {
+    FieldSpec.DataType dataType = columnMetadata.getDataType();
+    _columnToGlobalDictionary.putIfAbsent(column, new OrigAndDerivedValueHolder(dataType));
+    OrigAndDerivedValueHolder origAndDerivedValueHolder = _columnToGlobalDictionary.get(column);
+    if (dataType == FieldSpec.DataType.BYTES) {
+      origAndDerivedValueHolder.setOrigValue(new ByteArray((byte[])origValue));
+    } else {
+      origAndDerivedValueHolder.setOrigValue(origValue);
+    }
+  }
+
+  /**
+   * This is the second step where we complete the global dictionaries
+   * by sorting the original values to get sort order across all segments
+   */
+  @Override
+  public void sortOriginalValuesInGlobalDictionaries() {
+    // NO-OP since we use a sorted map so the global dictionary
+    // is already sorted
+  }
+
+  /**
+   * This is the third and final step where we complete the global
+   * dictionaries by generating values:
+   *
+   * For numeric columns, we generate in order since
+   * we start with a base value.
+   * For string column, we first generate and then sort
+   */
+  @Override
+  public void addDerivedValuesToGlobalDictionaries() {
+    // update global dictionary for each column by adding
+    // the corresponding generated value for each orig value
+    for (Map.Entry<String, OrigAndDerivedValueHolder> entry : _columnToGlobalDictionary.entrySet()) {
+      OrigAndDerivedValueHolder origAndDerivedValueHolder = entry.getValue();
+      generateDerivedValuesForGlobalDictionary(origAndDerivedValueHolder);
+    }
+  }
+
+  @Override
+  public void serialize(String outputDir) throws Exception {
+    // write global dictionary for each column
+    for (String column : _columnToGlobalDictionary.keySet()) {
+      PrintWriter dictionaryWriter = new PrintWriter(new BufferedWriter(new FileWriter(outputDir + "/" + column + DICT_FILE_EXTENSION)));
+      OrigAndDerivedValueHolder origAndDerivedValueHolder = _columnToGlobalDictionary.get(column);
+      Set<Map.Entry<Object, DerivedValue>> entries = origAndDerivedValueHolder._origAndDerivedValues.entrySet();
+      Iterator<Map.Entry<Object, DerivedValue>> sortedIterator = entries.iterator();
+      while (sortedIterator.hasNext()) {
+        Map.Entry<Object, DerivedValue> entry = sortedIterator.next();
+        dictionaryWriter.println(entry.getKey());
+        dictionaryWriter.println(entry.getValue()._derivedValue);
+      }
+      dictionaryWriter.flush();
+    }
+  }
+
+  @Override
+  public Object getDerivedValueForOrigValueSV(String column, Object origValue) {
+    OrigAndDerivedValueHolder origAndDerivedValueHolder = _columnToGlobalDictionary.get(column);
+    TreeMap<Object, DerivedValue> sortedMap = origAndDerivedValueHolder._origAndDerivedValues;
+    return sortedMap.get(origValue)._derivedValue;
+  }
+
+  @Override
+  public Object[] getDerivedValuesForOrigValuesMV(String column, Object[] origMultiValues) {
+    OrigAndDerivedValueHolder origAndDerivedValueHolder = _columnToGlobalDictionary.get(column);
+    TreeMap<Object, DerivedValue> sortedMap = origAndDerivedValueHolder._origAndDerivedValues;
+    int length = origMultiValues.length;
+    Object[] derivedMultiValues = new Object[length];
+    for (int i = 0; i < length; i++) {
+      derivedMultiValues[i] = sortedMap.get(origMultiValues[i]);
+    }
+    return derivedMultiValues;
+  }
+
+  private static class OrigAndDerivedValueHolder {
+    FieldSpec.DataType _dataType;
+    TreeMap<Object, DerivedValue> _origAndDerivedValues;
+
+    OrigAndDerivedValueHolder(FieldSpec.DataType dataType) {
+      _dataType = dataType;
+      _origAndDerivedValues = new TreeMap<>();
+    }
+
+    void setOrigValue(Object origValue) {
+     if (!_origAndDerivedValues.containsKey(origValue)) {
+       _origAndDerivedValues.put(origValue, new DerivedValue());
+     }
+    }
+  }
+
+  private static class DerivedValue {
 
 Review comment:
   This wrapper is needed so that we can set the derived values in global dictionary in sorted order while iterating over it

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5061: Improvements to data anonymizer tool

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #5061: Improvements to data anonymizer tool
URL: https://github.com/apache/incubator-pinot/pull/5061#discussion_r377274030
 
 

 ##########
 File path: pinot-tools/src/main/java/org/apache/pinot/tools/anonymizer/ArrayBasedGlobalDictionaries.java
 ##########
 @@ -0,0 +1,385 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.tools.anonymizer;
+
+import com.google.common.base.Preconditions;
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.PrintWriter;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Random;
+import org.apache.commons.lang.RandomStringUtils;
+import org.apache.pinot.core.segment.index.ColumnMetadata;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.ByteArray;
+
+
+public class ArrayBasedGlobalDictionaries implements GlobalDictionaries {
 
 Review comment:
   Nothing changed here from original code. Just moved to a new class (earlier this whole implementation was inside the PinotDataAndQueryAnonymizer class)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5061: Improvements to data anonymizer tool

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #5061: Improvements to data anonymizer tool
URL: https://github.com/apache/incubator-pinot/pull/5061#discussion_r377356752
 
 

 ##########
 File path: pinot-tools/src/main/java/org/apache/pinot/tools/anonymizer/MapBasedGlobalDictionaries.java
 ##########
 @@ -0,0 +1,278 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.tools.anonymizer;
+
+import java.io.BufferedWriter;
+import java.io.FileWriter;
+import java.io.PrintWriter;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.TreeMap;
+import org.apache.commons.lang.RandomStringUtils;
+import org.apache.pinot.core.segment.index.ColumnMetadata;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.ByteArray;
+
+
+public class MapBasedGlobalDictionaries implements GlobalDictionaries {
+  private final static int INT_BASE_VALUE = 1000;
+  private final static long LONG_BASE_VALUE = 100000;
+  private static final float FLOAT_BASE_VALUE = 100.23f;
+  private static final double DOUBLE_BASE_VALUE = 1000.2375;
+
+  private final Map<String, OrigAndDerivedValueHolder> _columnToGlobalDictionary;
+
+  MapBasedGlobalDictionaries() {
+    _columnToGlobalDictionary = new HashMap<>();
+  }
+
+  /**
+   * First step towards building global dictionary
+   * by inserting the original values from segments
+   * into global dictionary
+   * @param origValue original value
+   * @param column column name
+   * @param columnMetadata column metadata
+   * @param cardinality total cardinality of column
+   */
+  @Override
+  public void addOrigValueToGlobalDictionary(
+      Object origValue,
+      String column,
+      ColumnMetadata columnMetadata,
+      int cardinality) {
+    FieldSpec.DataType dataType = columnMetadata.getDataType();
+    _columnToGlobalDictionary.putIfAbsent(column, new OrigAndDerivedValueHolder(dataType));
+    OrigAndDerivedValueHolder origAndDerivedValueHolder = _columnToGlobalDictionary.get(column);
+    if (dataType == FieldSpec.DataType.BYTES) {
+      origAndDerivedValueHolder.setOrigValue(new ByteArray((byte[])origValue));
+    } else {
+      origAndDerivedValueHolder.setOrigValue(origValue);
+    }
+  }
+
+  /**
+   * This is the second step where we complete the global dictionaries
+   * by sorting the original values to get sort order across all segments
+   */
+  @Override
+  public void sortOriginalValuesInGlobalDictionaries() {
+    // NO-OP since we use a sorted map so the global dictionary
+    // is already sorted
+  }
+
+  /**
+   * This is the third and final step where we complete the global
+   * dictionaries by generating values:
+   *
+   * For numeric columns, we generate in order since
+   * we start with a base value.
+   * For string column, we first generate and then sort
+   */
+  @Override
+  public void addDerivedValuesToGlobalDictionaries() {
+    // update global dictionary for each column by adding
+    // the corresponding generated value for each orig value
+    for (Map.Entry<String, OrigAndDerivedValueHolder> entry : _columnToGlobalDictionary.entrySet()) {
+      OrigAndDerivedValueHolder origAndDerivedValueHolder = entry.getValue();
+      generateDerivedValuesForGlobalDictionary(origAndDerivedValueHolder);
+    }
+  }
+
+  @Override
+  public void serialize(String outputDir) throws Exception {
+    // write global dictionary for each column
+    for (String column : _columnToGlobalDictionary.keySet()) {
+      PrintWriter dictionaryWriter = new PrintWriter(new BufferedWriter(new FileWriter(outputDir + "/" + column + DICT_FILE_EXTENSION)));
+      OrigAndDerivedValueHolder origAndDerivedValueHolder = _columnToGlobalDictionary.get(column);
+      Set<Map.Entry<Object, DerivedValue>> entries = origAndDerivedValueHolder._origAndDerivedValues.entrySet();
+      Iterator<Map.Entry<Object, DerivedValue>> sortedIterator = entries.iterator();
+      while (sortedIterator.hasNext()) {
+        Map.Entry<Object, DerivedValue> entry = sortedIterator.next();
+        dictionaryWriter.println(entry.getKey());
+        dictionaryWriter.println(entry.getValue()._derivedValue);
+      }
+      dictionaryWriter.flush();
+    }
+  }
+
+  @Override
+  public Object getDerivedValueForOrigValueSV(String column, Object origValue) {
+    OrigAndDerivedValueHolder origAndDerivedValueHolder = _columnToGlobalDictionary.get(column);
+    TreeMap<Object, DerivedValue> sortedMap = origAndDerivedValueHolder._origAndDerivedValues;
+    return sortedMap.get(origValue)._derivedValue;
+  }
+
+  @Override
+  public Object[] getDerivedValuesForOrigValuesMV(String column, Object[] origMultiValues) {
+    OrigAndDerivedValueHolder origAndDerivedValueHolder = _columnToGlobalDictionary.get(column);
+    TreeMap<Object, DerivedValue> sortedMap = origAndDerivedValueHolder._origAndDerivedValues;
+    int length = origMultiValues.length;
+    Object[] derivedMultiValues = new Object[length];
+    for (int i = 0; i < length; i++) {
+      derivedMultiValues[i] = sortedMap.get(origMultiValues[i]);
+    }
+    return derivedMultiValues;
+  }
+
+  private static class OrigAndDerivedValueHolder {
+    FieldSpec.DataType _dataType;
+    TreeMap<Object, DerivedValue> _origAndDerivedValues;
+
+    OrigAndDerivedValueHolder(FieldSpec.DataType dataType) {
+      _dataType = dataType;
+      _origAndDerivedValues = new TreeMap<>();
+    }
+
+    void setOrigValue(Object origValue) {
+     if (!_origAndDerivedValues.containsKey(origValue)) {
+       _origAndDerivedValues.put(origValue, new DerivedValue());
+     }
+    }
+  }
+
+  private static class DerivedValue {
 
 Review comment:
   Done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #5061: Improvements to data anonymizer tool

Posted by GitBox <gi...@apache.org>.
jackjlli commented on a change in pull request #5061: Improvements to data anonymizer tool
URL: https://github.com/apache/incubator-pinot/pull/5061#discussion_r377309181
 
 

 ##########
 File path: pinot-tools/src/main/java/org/apache/pinot/tools/anonymizer/MapBasedGlobalDictionaries.java
 ##########
 @@ -0,0 +1,278 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.tools.anonymizer;
+
+import java.io.BufferedWriter;
+import java.io.FileWriter;
+import java.io.PrintWriter;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.TreeMap;
+import org.apache.commons.lang.RandomStringUtils;
+import org.apache.pinot.core.segment.index.ColumnMetadata;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.ByteArray;
+
+
+public class MapBasedGlobalDictionaries implements GlobalDictionaries {
+  private final static int INT_BASE_VALUE = 1000;
+  private final static long LONG_BASE_VALUE = 100000;
+  private static final float FLOAT_BASE_VALUE = 100.23f;
+  private static final double DOUBLE_BASE_VALUE = 1000.2375;
+
+  private final Map<String, OrigAndDerivedValueHolder> _columnToGlobalDictionary;
+
+  MapBasedGlobalDictionaries() {
+    _columnToGlobalDictionary = new HashMap<>();
+  }
+
+  /**
+   * First step towards building global dictionary
+   * by inserting the original values from segments
+   * into global dictionary
+   * @param origValue original value
+   * @param column column name
+   * @param columnMetadata column metadata
+   * @param cardinality total cardinality of column
+   */
+  @Override
+  public void addOrigValueToGlobalDictionary(
+      Object origValue,
+      String column,
+      ColumnMetadata columnMetadata,
+      int cardinality) {
+    FieldSpec.DataType dataType = columnMetadata.getDataType();
+    _columnToGlobalDictionary.putIfAbsent(column, new OrigAndDerivedValueHolder(dataType));
+    OrigAndDerivedValueHolder origAndDerivedValueHolder = _columnToGlobalDictionary.get(column);
+    if (dataType == FieldSpec.DataType.BYTES) {
+      origAndDerivedValueHolder.setOrigValue(new ByteArray((byte[])origValue));
+    } else {
+      origAndDerivedValueHolder.setOrigValue(origValue);
+    }
+  }
+
+  /**
+   * This is the second step where we complete the global dictionaries
+   * by sorting the original values to get sort order across all segments
+   */
+  @Override
+  public void sortOriginalValuesInGlobalDictionaries() {
+    // NO-OP since we use a sorted map so the global dictionary
+    // is already sorted
+  }
+
+  /**
+   * This is the third and final step where we complete the global
+   * dictionaries by generating values:
+   *
+   * For numeric columns, we generate in order since
+   * we start with a base value.
+   * For string column, we first generate and then sort
+   */
+  @Override
+  public void addDerivedValuesToGlobalDictionaries() {
+    // update global dictionary for each column by adding
+    // the corresponding generated value for each orig value
+    for (Map.Entry<String, OrigAndDerivedValueHolder> entry : _columnToGlobalDictionary.entrySet()) {
+      OrigAndDerivedValueHolder origAndDerivedValueHolder = entry.getValue();
+      generateDerivedValuesForGlobalDictionary(origAndDerivedValueHolder);
+    }
+  }
+
+  @Override
+  public void serialize(String outputDir) throws Exception {
+    // write global dictionary for each column
+    for (String column : _columnToGlobalDictionary.keySet()) {
+      PrintWriter dictionaryWriter = new PrintWriter(new BufferedWriter(new FileWriter(outputDir + "/" + column + DICT_FILE_EXTENSION)));
+      OrigAndDerivedValueHolder origAndDerivedValueHolder = _columnToGlobalDictionary.get(column);
+      Set<Map.Entry<Object, DerivedValue>> entries = origAndDerivedValueHolder._origAndDerivedValues.entrySet();
+      Iterator<Map.Entry<Object, DerivedValue>> sortedIterator = entries.iterator();
+      while (sortedIterator.hasNext()) {
+        Map.Entry<Object, DerivedValue> entry = sortedIterator.next();
+        dictionaryWriter.println(entry.getKey());
+        dictionaryWriter.println(entry.getValue()._derivedValue);
+      }
+      dictionaryWriter.flush();
+    }
+  }
+
+  @Override
+  public Object getDerivedValueForOrigValueSV(String column, Object origValue) {
+    OrigAndDerivedValueHolder origAndDerivedValueHolder = _columnToGlobalDictionary.get(column);
+    TreeMap<Object, DerivedValue> sortedMap = origAndDerivedValueHolder._origAndDerivedValues;
+    return sortedMap.get(origValue)._derivedValue;
+  }
+
+  @Override
+  public Object[] getDerivedValuesForOrigValuesMV(String column, Object[] origMultiValues) {
+    OrigAndDerivedValueHolder origAndDerivedValueHolder = _columnToGlobalDictionary.get(column);
+    TreeMap<Object, DerivedValue> sortedMap = origAndDerivedValueHolder._origAndDerivedValues;
+    int length = origMultiValues.length;
+    Object[] derivedMultiValues = new Object[length];
+    for (int i = 0; i < length; i++) {
+      derivedMultiValues[i] = sortedMap.get(origMultiValues[i]);
+    }
+    return derivedMultiValues;
+  }
+
+  private static class OrigAndDerivedValueHolder {
+    FieldSpec.DataType _dataType;
+    TreeMap<Object, DerivedValue> _origAndDerivedValues;
+
+    OrigAndDerivedValueHolder(FieldSpec.DataType dataType) {
+      _dataType = dataType;
+      _origAndDerivedValues = new TreeMap<>();
+    }
+
+    void setOrigValue(Object origValue) {
+     if (!_origAndDerivedValues.containsKey(origValue)) {
+       _origAndDerivedValues.put(origValue, new DerivedValue());
+     }
+    }
+  }
+
+  private static class DerivedValue {
 
 Review comment:
   Can you put this comment on top of this class?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #5061: Improvements to data anonymizer tool

Posted by GitBox <gi...@apache.org>.
jackjlli commented on a change in pull request #5061: Improvements to data anonymizer tool
URL: https://github.com/apache/incubator-pinot/pull/5061#discussion_r377273727
 
 

 ##########
 File path: pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/AnonymizeDataCommand.java
 ##########
 @@ -116,7 +116,8 @@ public boolean execute()
           _outputDir,
           _avroFileNamePrefix,
           filterColumnCardinalityMap,
-          columnsToRetainDataFor);
+          columnsToRetainDataFor,
+          true);
 
 Review comment:
   You can make it configurable instead of hard coding it as true.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-io edited a comment on issue #5061: Improvements to data anonymizer tool

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #5061: Improvements to data anonymizer tool
URL: https://github.com/apache/incubator-pinot/pull/5061#issuecomment-584332757
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5061?src=pr&el=h1) Report
   > Merging [#5061](https://codecov.io/gh/apache/incubator-pinot/pull/5061?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/f5fe4af026824c2096559c62dfaac3ef52542b79?src=pr&el=desc) will **increase** coverage by `18.76%`.
   > The diff coverage is `0.31%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/5061/graphs/tree.svg?width=650&token=4ibza2ugkz&height=150&src=pr)](https://codecov.io/gh/apache/incubator-pinot/pull/5061?src=pr&el=tree)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #5061       +/-   ##
   =============================================
   + Coverage     39.63%   58.39%   +18.76%     
   - Complexity        0       12       +12     
   =============================================
     Files          1182     1185        +3     
     Lines         63135    63394      +259     
     Branches       9287     9315       +28     
   =============================================
   + Hits          25022    37020    +11998     
   + Misses        35935    23659    -12276     
   - Partials       2178     2715      +537
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5061?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...inot/tools/admin/command/AnonymizeDataCommand.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtdG9vbHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3Rvb2xzL2FkbWluL2NvbW1hbmQvQW5vbnltaXplRGF0YUNvbW1hbmQuamF2YQ==) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...t/tools/anonymizer/MapBasedGlobalDictionaries.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtdG9vbHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3Rvb2xzL2Fub255bWl6ZXIvTWFwQmFzZWRHbG9iYWxEaWN0aW9uYXJpZXMuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | |
   | [...tools/anonymizer/ArrayBasedGlobalDictionaries.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtdG9vbHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3Rvb2xzL2Fub255bWl6ZXIvQXJyYXlCYXNlZEdsb2JhbERpY3Rpb25hcmllcy5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | |
   | [.../tools/anonymizer/PinotDataAndQueryAnonymizer.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtdG9vbHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3Rvb2xzL2Fub255bWl6ZXIvUGlub3REYXRhQW5kUXVlcnlBbm9ueW1pemVyLmphdmE=) | `41.05% <3.57%> (ø)` | `0 <0> (?)` | |
   | [...ller/validation/OfflineSegmentIntervalChecker.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL09mZmxpbmVTZWdtZW50SW50ZXJ2YWxDaGVja2VyLmphdmE=) | `32.05% <0%> (-48.72%)` | `0% <0%> (ø)` | |
   | [.../org/apache/pinot/client/PinotClientException.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1Bpbm90Q2xpZW50RXhjZXB0aW9uLmphdmE=) | `33.33% <0%> (-33.34%)` | `0% <0%> (ø)` | |
   | [...apache/pinot/common/metrics/ValidationMetrics.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9WYWxpZGF0aW9uTWV0cmljcy5qYXZh) | `47.45% <0%> (-28.82%)` | `0% <0%> (ø)` | |
   | [...inot/client/JsonAsyncHttpPinotClientTransport.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0pzb25Bc3luY0h0dHBQaW5vdENsaWVudFRyYW5zcG9ydC5qYXZh) | `62% <0%> (-14%)` | `0% <0%> (ø)` | |
   | [...core/operator/dociditerators/AndDocIdIterator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9BbmREb2NJZEl0ZXJhdG9yLmphdmE=) | `77.58% <0%> (-10.35%)` | `0% <0%> (ø)` | |
   | [...pinot/broker/api/resources/PinotClientRequest.java](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdENsaWVudFJlcXVlc3QuamF2YQ==) | `23.4% <0%> (-8.52%)` | `0% <0%> (ø)` | |
   | ... and [578 more](https://codecov.io/gh/apache/incubator-pinot/pull/5061/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5061?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5061?src=pr&el=footer). Last update [f5fe4af...7f60204](https://codecov.io/gh/apache/incubator-pinot/pull/5061?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5061: Improvements to data anonymizer tool

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #5061: Improvements to data anonymizer tool
URL: https://github.com/apache/incubator-pinot/pull/5061#discussion_r377356820
 
 

 ##########
 File path: pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/AnonymizeDataCommand.java
 ##########
 @@ -116,7 +116,8 @@ public boolean execute()
           _outputDir,
           _avroFileNamePrefix,
           filterColumnCardinalityMap,
-          columnsToRetainDataFor);
+          columnsToRetainDataFor,
+          true);
 
 Review comment:
   Done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org