You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datasketches.apache.org by GitBox <gi...@apache.org> on 2022/01/05 20:48:53 UTC

[GitHub] [datasketches-java] will-lauer commented on a change in pull request #379: Add heapify wrap compact sketch2

will-lauer commented on a change in pull request #379:
URL: https://github.com/apache/datasketches-java/pull/379#discussion_r779120696



##########
File path: src/main/java/org/apache/datasketches/theta/SingleItemSketch.java
##########
@@ -99,6 +101,21 @@ public static SingleItemSketch heapify(final Memory srcMem, final long seed) {
     throw new SketchesArgumentException("Input Memory is not a SingleItemSketch.");
   }
 
+  /**
+   * Creates a SingleItemSketch on the heap given a SingleItemSketch Memory image and a seedHash.
+   * Checks the seed hash of the given Memory against a hash of the given seed.
+   * @param srcMem the Memory to be heapified.
+   * @param seedHash a given seedHash
+   * @return a SingleItemSketch
+   */ //does not override Sketch
+  public static SingleItemSketch heapify(final Memory srcMem, final short seedHash) {

Review comment:
       it would probably be clearer here if `seedHash` was renamed `expectedSeedHash` as the actual seed has is extracted from the `srcMem` object.

##########
File path: src/main/java/org/apache/datasketches/theta/CompactSketch.java
##########
@@ -19,25 +19,241 @@
 
 package org.apache.datasketches.theta;
 
+import static org.apache.datasketches.Family.idToFamily;
+import static org.apache.datasketches.Util.DEFAULT_UPDATE_SEED;
+import static org.apache.datasketches.theta.PreambleUtil.COMPACT_FLAG_MASK;
+import static org.apache.datasketches.theta.PreambleUtil.EMPTY_FLAG_MASK;
+import static org.apache.datasketches.theta.PreambleUtil.FAMILY_BYTE;
+import static org.apache.datasketches.theta.PreambleUtil.FLAGS_BYTE;
+import static org.apache.datasketches.theta.PreambleUtil.ORDERED_FLAG_MASK;
+import static org.apache.datasketches.theta.PreambleUtil.READ_ONLY_FLAG_MASK;
+import static org.apache.datasketches.theta.PreambleUtil.SER_VER_BYTE;
+import static org.apache.datasketches.theta.PreambleUtil.extractSeedHash;
+import static org.apache.datasketches.theta.SingleItemSketch.otherCheckForSingleItem;
+
 import org.apache.datasketches.Family;
+import org.apache.datasketches.SketchesArgumentException;
+import org.apache.datasketches.Util;
+import org.apache.datasketches.memory.Memory;
 import org.apache.datasketches.memory.WritableMemory;
 
 /**
  * The parent class of all the CompactSketches. CompactSketches are never created directly.
- * They are created as a result of the compact() method of an UpdateSketch or as a result of a
- * getResult() of a SetOperation.
+ * They are created as a result of the compact() method of an UpdateSketch, a result of a
+ * getResult() of a SetOperation, or from a heapify method.
  *
  * <p>A CompactSketch is the simplest form of a Theta Sketch. It consists of a compact list
  * (i.e., no intervening spaces) of hash values, which may be ordered or not, a value for theta
- * and a seed hash.  A CompactSketch is read-only,
+ * and a seed hash.  A CompactSketch is immutable (read-only),
  * and the space required when stored is only the space required for the hash values and 8 to 24
  * bytes of preamble. An empty CompactSketch consumes only 8 bytes.</p>
  *
  * @author Lee Rhodes
  */
 public abstract class CompactSketch extends Sketch {
+  private static final short defaultSeedHash = Util.computeSeedHash(DEFAULT_UPDATE_SEED);
+
+  /**
+   * Heapify takes a CompactSketch image in Memory and instantiates an on-heap CompactSketch.
+   *
+   * <p>The resulting sketch will not retain any link to the source Memory.</p>
+   *
+   * <p>This method assumes that the sketch image was created with the correct hash seed,
+   * so it is not checked.</p>
+   *
+   * @param srcMem an image of a CompactSketch.
+   * <a href="{@docRoot}/resources/dictionary.html#mem">See Memory</a>.
+   * @return a CompactSketch on the heap.
+   */
+  public static CompactSketch heapify(final Memory srcMem) {
+    final int serVer = srcMem.getByte(SER_VER_BYTE) & 0XFF;
+    final int familyID = srcMem.getByte(FAMILY_BYTE) & 0XFF;
+    final Family family = Family.idToFamily(familyID);
+    if (family != Family.COMPACT) {
+      throw new IllegalArgumentException("Corrupted: " + family + " is not Compact!");
+    }
+    if (serVer == 3) { //no seed check
+      final int flags = PreambleUtil.extractFlags(srcMem);
+      final boolean srcOrdered = (flags & ORDERED_FLAG_MASK) != 0;
+      return CompactOperations.memoryToCompact(srcMem, srcOrdered, null);
+    }
+    //not SerVer 3, assume compact stored form
+    if (serVer == 1) {
+      return ForwardCompatibility.heapify1to3(srcMem, defaultSeedHash);
+    }
+    if (serVer == 2) {
+      final short srcSeedHash = (short) extractSeedHash(srcMem);
+      return ForwardCompatibility.heapify2to3(srcMem, srcSeedHash);
+    }
+    throw new SketchesArgumentException("Unknown Serialization Version: " + serVer);
+  }
+
+  /**
+   * Heapify takes a CompactSketch image in Memory and instantiates an on-heap CompactSketch.
+   *
+   * <p>The resulting sketch will not retain any link to the source Memory.</p>
+   *
+   * <p>This this method checks if the given seed was used to create the source Memory image.
+   * However, SerialVersion 1 sketches cannot be checked.</p>
+   *
+   * @param srcMem an image of a CompactSketch that was created using the given seed.
+   * <a href="{@docRoot}/resources/dictionary.html#mem">See Memory</a>.
+   * @param seed <a href="{@docRoot}/resources/dictionary.html#seed">See Update Hash Seed</a>.
+   * @return a CompactSketch on the heap.
+   */
+  public static CompactSketch heapify(final Memory srcMem, final long seed) {

Review comment:
       we should think about renaming `seed` here to `expectedSeed`, as mostly seed is being used here as a validation of the seedHash that is in the `srcMem`, and not setting the seed to be a specific value (v1 sketches excepted). Even though I know what this was supposed to do, I got confused reading the code because I was thinking this was passing in the seed  rather than specifying the expected value to validate against.

##########
File path: src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java
##########
@@ -60,16 +60,16 @@
    * Wraps the given Memory, which must be a SerVer 3, ordered, CompactSketch image.
    * Must check the validity of the Memory before calling. The order bit must be set properly.
    * @param srcMem <a href="{@docRoot}/resources/dictionary.html#mem">See Memory</a>
-   * @param seed The update seed.
-   * <a href="{@docRoot}/resources/dictionary.html#seed">See Update Hash Seed</a>.
+   * @param seedHash The update seedHash.
+   * <a href="{@docRoot}/resources/dictionary.html#seedHash">See Seed Hash</a>.
    * @return this sketch
    */
-  static DirectCompactSketch wrapInstance(final Memory srcMem, final long seed) {
-    checkMemorySeedHash(srcMem, seed);
+  static DirectCompactSketch wrapInstance(final Memory srcMem, final short seedHash) {

Review comment:
       `seedHash` here should probably be renamed to `expectedSeedHash` to clarify its use.

##########
File path: src/main/java/org/apache/datasketches/theta/Sketch.java
##########
@@ -57,125 +53,140 @@
   //public static factory constructor-type methods
 
   /**
-   * Heapify takes the sketch image in Memory and instantiates an on-heap
-   * Sketch using the
-   * <a href="{@docRoot}/resources/dictionary.html#defaultUpdateSeed">Default Update Seed</a>.
-   * The resulting sketch will not retain any link to the source Memory.
-   * @param srcMem an image of a Sketch where the image seed hash matches the default seed hash.
-   * <a href="{@docRoot}/resources/dictionary.html#mem">See Memory</a>
-   * @return a Heap-based Sketch from the given Memory
+   * Heapify takes the sketch image in Memory and instantiates an on-heap Sketch.
+   *
+   * <p>The resulting sketch will not retain any link to the source Memory.</p>
+   *
+   * <p>For Update Sketches this method checks if the
+   * <a href="{@docRoot}/resources/dictionary.html#defaultUpdateSeed">Default Update Seed</a></p>
+   * was used to create the source Memory image.
+   *
+   * <p>For Compact Sketches this method assumes that the sketch image was created with the
+   * correct hash seed, so it is not checked.</p>
+   *
+   * @param srcMem an image of a Sketch.
+   * <a href="{@docRoot}/resources/dictionary.html#mem">See Memory</a>.
+   * @return a Sketch on the heap.
    */
   public static Sketch heapify(final Memory srcMem) {
-    return heapify(srcMem, DEFAULT_UPDATE_SEED);
+    final byte familyID = srcMem.getByte(FAMILY_BYTE);
+    final Family family = idToFamily(familyID);
+    if (family == Family.COMPACT) {
+      return CompactSketch.heapify(srcMem);
+    }
+    return heapifyUpdateFromMemory(srcMem, DEFAULT_UPDATE_SEED);
   }
 
   /**
-   * Heapify takes the sketch image in Memory and instantiates an on-heap
-   * Sketch using the given seed.
-   * The resulting sketch will not retain any link to the source Memory.
-   * @param srcMem an image of a Sketch where the image seed hash matches the given seed hash.
-   * <a href="{@docRoot}/resources/dictionary.html#mem">See Memory</a>
+   * Heapify takes the sketch image in Memory and instantiates an on-heap Sketch.
+   *
+   * <p>The resulting sketch will not retain any link to the source Memory.</p>
+   *
+   * <p>For Update and Compact Sketches this method checks if the given seed was used to
+   * create the source Memory image.  However, SerialVersion 1 sketches cannot be checked.</p>
+   *
+   * @param srcMem an image of a Sketch that was created using the given seed.
+   * <a href="{@docRoot}/resources/dictionary.html#mem">See Memory</a>.
    * @param seed <a href="{@docRoot}/resources/dictionary.html#seed">See Update Hash Seed</a>.
    * Compact sketches store a 16-bit hash of the seed, but not the seed itself.
-   * @return a Heap-based Sketch from the given Memory
+   * @return a Sketch on the heap.
    */
   public static Sketch heapify(final Memory srcMem, final long seed) {
-    final int serVer = srcMem.getByte(SER_VER_BYTE);
-    if (serVer == 3) {
-      return heapifyFromMemory(srcMem, seed);
-    }
-    if (serVer == 1) {
-      return ForwardCompatibility.heapify1to3(srcMem, seed);
-    }
-    if (serVer == 2) {
-      return ForwardCompatibility.heapify2to3(srcMem, seed);
+    final byte familyID = srcMem.getByte(FAMILY_BYTE);
+    final Family family = idToFamily(familyID);
+    if (family == Family.COMPACT) {
+      return CompactSketch.heapify(srcMem, seed);
     }
-    throw new SketchesArgumentException("Unknown Serialization Version: " + serVer);
+    return heapifyUpdateFromMemory(srcMem, seed);
   }
 
   /**
-   * Wrap takes the sketch image in Memory and refers to it directly. There is no data copying onto
-   * the java heap.  Only "Direct" Serialization Version 3 (i.e, OpenSource) sketches that have
-   * been explicitly stored as direct objects can be wrapped. This method assumes the
-   * {@link org.apache.datasketches.Util#DEFAULT_UPDATE_SEED}.
-   * <a href="{@docRoot}/resources/dictionary.html#defaultUpdateSeed">Default Update Seed</a>.
-   * @param srcMem an image of a Sketch where the image seed hash matches the default seed hash.
-   * <a href="{@docRoot}/resources/dictionary.html#mem">See Memory</a>
+   * Wrap takes the sketch image in the given Memory and refers to it directly.
+   * There is no data copying onto the java heap.
+   * The wrap operation enables fast read-only merging and access to all the public read-only API.
+   *
+   * <p>Only "Direct" Serialization Version 3 (i.e, OpenSource) sketches that have
+   * been explicitly stored as direct sketches can be wrapped.
+   * Wrapping earlier serial version sketches will result in a on-heap CompactSketch
+   * where all data will be copied to the heap. These early versions were never designed to
+   * "wrap".</p>
+   *
+   * <p>Wrapping any subclass of this class that is empty or contains only a single item will
+   * result in on-heap equivalent forms of empty and single item sketch respectively.
+   * This is actually faster and consumes less overall memory.</p>
+   *
+   * <p>For Update Sketches this method checks if the
+   * <a href="{@docRoot}/resources/dictionary.html#defaultUpdateSeed">Default Update Seed</a></p>
+   * was used to create the source Memory image.
+   *
+   * <p>For Compact Sketches this method assumes that the sketch image was created with the
+   * correct hash seed, so it is not checked.</p>
+   *
+   * @param srcMem an image of a Sketch.
+   * <a href="{@docRoot}/resources/dictionary.html#mem">See Memory</a>.
    * @return a Sketch backed by the given Memory
    */
   public static Sketch wrap(final Memory srcMem) {
-    return wrap(srcMem, DEFAULT_UPDATE_SEED);
+    final int  preLongs = srcMem.getByte(PREAMBLE_LONGS_BYTE) & 0X3F;
+    final int serVer = srcMem.getByte(SER_VER_BYTE) & 0XFF;
+    final int familyID = srcMem.getByte(FAMILY_BYTE) & 0XFF;
+    final Family family = Family.idToFamily(familyID);
+    if (family == Family.QUICKSELECT) {

Review comment:
       Is it possible to have family == ALPHA here?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org