You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datasketches.apache.org by le...@apache.org on 2020/07/07 23:37:03 UTC
[incubator-datasketches-java] branch Refactor_Theta_Tuple updated
(bc77a08 -> 644ab85)
This is an automated email from the ASF dual-hosted git repository.
leerho pushed a change to branch Refactor_Theta_Tuple
in repository https://gitbox.apache.org/repos/asf/incubator-datasketches-java.git.
from bc77a08 Interim changes, all tests work.
new 85d4442 Interim. This removed much of the code in the PairwiseSetOperations class and replaced it with references to the now equivalent classes that extend SetOperation.
new f421f79 Eliminated two Classes: DirectCompactOrderedSketch and DirectCompactUnorderedSketch.
new f4317e5 Consolidated all compacting operations into CompactOperations.
new 184623e Interim with errors
new 50f49bf Deprecated getCurrentBytes(boolean). Replaced by getCompactBytes() and getCurrentBytes().
new 8736b03 Interm: Completed major rewrite of Concurrent Theta Sketch unit tests and cleaned up main code as well.
add 206cb81 Fix one-off bug that Alex found.
add 29bb055 More of Alex's bug
add ca5f0da Merge pull request #322 from apache/Alex's_Bug
add fd4c995 Small changes needed in master before the next set of PRs.
add a22af6a Add assert to test.
add eead07c Merge pull request #323 from apache/SmallCleanup
new 46746a8 Merge branch 'master' into Refactor_Theta_Tuple
new 65897ed Allow for future extension of Theta.
new e3f4ad2 Close to final set of changes.
new 644ab85 Add unit tests
The 10 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/datasketches/HashOperations.java | 193 ++--
.../java/org/apache/datasketches/theta/AnotB.java | 31 +-
.../org/apache/datasketches/theta/AnotBimpl.java | 327 +-----
.../datasketches/theta/CompactOperations.java | 123 ++-
.../apache/datasketches/theta/CompactSketch.java | 10 +
.../theta/ConcurrentHeapThetaBuffer.java | 104 +-
.../theta/ConcurrentSharedThetaSketch.java | 19 +-
.../theta/DirectCompactOrderedSketch.java | 60 -
.../datasketches/theta/DirectCompactSketch.java | 53 +-
.../theta/DirectCompactUnorderedSketch.java | 60 -
.../theta/DirectQuickSelectSketch.java | 12 +-
.../theta/DirectQuickSelectSketchR.java | 59 +-
.../datasketches/theta/EmptyCompactSketch.java | 11 +-
.../datasketches/theta/ForwardCompatibility.java | 6 +-
.../apache/datasketches/theta/HeapAlphaSketch.java | 11 +-
.../theta/HeapCompactOrderedSketch.java | 70 --
.../datasketches/theta/HeapCompactSketch.java | 76 +-
.../theta/HeapCompactUnorderedSketch.java | 70 --
.../datasketches/theta/HeapQuickSelectSketch.java | 34 +-
.../datasketches/theta/HeapUpdateSketch.java | 28 +-
.../datasketches/theta/IntersectionImpl.java | 8 +-
.../datasketches/theta/IntersectionImplR.java | 19 +-
.../datasketches/theta/JaccardSimilarity.java | 12 +-
.../datasketches/theta/PairwiseSetOperations.java | 156 +--
.../apache/datasketches/theta/SetOperation.java | 18 +-
.../datasketches/theta/SingleItemSketch.java | 11 +-
.../java/org/apache/datasketches/theta/Sketch.java | 123 +--
.../datasketches/theta/UpdateReturnState.java | 26 +-
.../apache/datasketches/theta/UpdateSketch.java | 140 +--
.../datasketches/theta/UpdateSketchBuilder.java | 6 +-
.../java/org/apache/datasketches/tuple/AnotB.java | 4 +-
.../apache/datasketches/tuple/Intersection.java | 6 +-
.../datasketches/tuple/QuickSelectSketch.java | 41 +-
.../ArrayOfDoublesQuickSelectSketch.java | 4 +-
.../DirectArrayOfDoublesQuickSelectSketch.java | 6 +-
.../apache/datasketches/HashOperationsTest.java | 23 +-
.../apache/datasketches/theta/AnotBimplTest.java | 21 +-
.../datasketches/theta/BackwardConversions.java | 4 +-
.../datasketches/theta/CompactSketchTest.java | 71 +-
.../ConcurrentDirectQuickSelectSketchTest.java | 1142 +++++++++-----------
.../theta/ConcurrentHeapQuickSelectSketchTest.java | 500 ++++-----
.../datasketches/theta/DirectIntersectionTest.java | 15 +-
.../theta/DirectQuickSelectSketchTest.java | 20 +-
.../apache/datasketches/theta/DirectUnionTest.java | 23 +-
.../org/apache/datasketches/theta/EmptyTest.java | 15 +-
.../theta/ForwardCompatibilityTest.java | 2 +-
.../datasketches/theta/HeapAlphaSketchTest.java | 28 +-
.../datasketches/theta/HeapIntersectionTest.java | 19 +-
.../theta/HeapQuickSelectSketchTest.java | 30 +-
.../apache/datasketches/theta/HeapUnionTest.java | 19 +-
.../datasketches/theta/SetOpsCornerCasesTest.java | 60 +-
.../datasketches/theta/SingleItemSketchTest.java | 25 +-
.../org/apache/datasketches/theta/SketchTest.java | 68 +-
.../apache/datasketches/theta/UnionImplTest.java | 8 +-
.../datasketches/theta/UpdateSketchTest.java | 13 +-
.../tuple/adouble/AdoubleAnotBTest.java | 2 +-
.../tuple/aninteger/IntegerSketchTest.java | 11 +-
57 files changed, 1696 insertions(+), 2360 deletions(-)
delete mode 100644 src/main/java/org/apache/datasketches/theta/DirectCompactOrderedSketch.java
delete mode 100644 src/main/java/org/apache/datasketches/theta/DirectCompactUnorderedSketch.java
delete mode 100644 src/main/java/org/apache/datasketches/theta/HeapCompactOrderedSketch.java
delete mode 100644 src/main/java/org/apache/datasketches/theta/HeapCompactUnorderedSketch.java
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[incubator-datasketches-java] 08/10: Allow for future extension of
Theta.
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leerho pushed a commit to branch Refactor_Theta_Tuple
in repository https://gitbox.apache.org/repos/asf/incubator-datasketches-java.git
commit 65897ed314bc9911e43d8aa64c06c62894f29337
Author: Lee Rhodes <le...@users.noreply.github.com>
AuthorDate: Sat Jul 4 11:47:52 2020 -0700
Allow for future extension of Theta.
---
.../theta/DirectQuickSelectSketch.java | 8 +++++--
.../apache/datasketches/theta/HeapAlphaSketch.java | 1 -
.../datasketches/theta/HeapQuickSelectSketch.java | 11 +++++----
.../datasketches/theta/UpdateReturnState.java | 26 ++++++++++++++++------
4 files changed, 32 insertions(+), 14 deletions(-)
diff --git a/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketch.java b/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketch.java
index 9309194..dc34bc5 100644
--- a/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketch.java
@@ -52,6 +52,8 @@ import static org.apache.datasketches.theta.Rebuilder.moveAndResize;
import static org.apache.datasketches.theta.Rebuilder.quickSelectAndRebuild;
import static org.apache.datasketches.theta.Rebuilder.resize;
import static org.apache.datasketches.theta.UpdateReturnState.InsertedCountIncremented;
+import static org.apache.datasketches.theta.UpdateReturnState.InsertedCountIncrementedRebuilt;
+import static org.apache.datasketches.theta.UpdateReturnState.InsertedCountIncrementedResized;
import static org.apache.datasketches.theta.UpdateReturnState.RejectedDuplicate;
import static org.apache.datasketches.theta.UpdateReturnState.RejectedOverTheta;
@@ -274,6 +276,7 @@ class DirectQuickSelectSketch extends DirectQuickSelectSketchR {
: "lgArr: " + lgArrLongs + ", lgNom: " + lgNomLongs;
//rebuild, refresh curCount based on # values in the hashtable.
quickSelectAndRebuild(wmem_, preambleLongs, lgNomLongs);
+ return InsertedCountIncrementedRebuilt;
} //end of rebuild, exit
else { //Not at full size, resize. Should not get here if lgRF = 0 and memCap is too small.
@@ -285,6 +288,7 @@ class DirectQuickSelectSketch extends DirectQuickSelectSketchR {
//lgArrLongs will change; thetaLong, curCount will not
resize(wmem_, preambleLongs, lgArrLongs, tgtLgArrLongs);
hashTableThreshold_ = setHashTableThreshold(lgNomLongs, tgtLgArrLongs);
+ return InsertedCountIncrementedResized;
} //end of Expand in current memory, exit.
else {
@@ -304,10 +308,10 @@ class DirectQuickSelectSketch extends DirectQuickSelectSketchR {
wmem_ = newDstMem;
hashTableThreshold_ = setHashTableThreshold(lgNomLongs, tgtLgArrLongs);
-
+ return InsertedCountIncrementedResized;
} //end of Request more memory to resize
} //end of resize
- }
+ } //end of isOutOfSpace
return InsertedCountIncremented;
}
diff --git a/src/main/java/org/apache/datasketches/theta/HeapAlphaSketch.java b/src/main/java/org/apache/datasketches/theta/HeapAlphaSketch.java
index 2561b44..b0782ec 100644
--- a/src/main/java/org/apache/datasketches/theta/HeapAlphaSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/HeapAlphaSketch.java
@@ -308,7 +308,6 @@ final class HeapAlphaSketch extends HeapUpdateSketch {
//The over-theta test
if (HashOperations.continueCondition(thetaLong_, hash)) {
- // very very unlikely that hash == Long.MAX_VALUE. It is ignored just as zero is ignored.
return RejectedOverTheta; //signal that hash was rejected due to theta.
}
diff --git a/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java b/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java
index d6303bf..b3dcf6b 100644
--- a/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java
@@ -36,6 +36,8 @@ import static org.apache.datasketches.theta.PreambleUtil.extractP;
import static org.apache.datasketches.theta.PreambleUtil.extractPreLongs;
import static org.apache.datasketches.theta.PreambleUtil.extractThetaLong;
import static org.apache.datasketches.theta.UpdateReturnState.InsertedCountIncremented;
+import static org.apache.datasketches.theta.UpdateReturnState.InsertedCountIncrementedRebuilt;
+import static org.apache.datasketches.theta.UpdateReturnState.InsertedCountIncrementedResized;
import static org.apache.datasketches.theta.UpdateReturnState.RejectedDuplicate;
import static org.apache.datasketches.theta.UpdateReturnState.RejectedOverTheta;
@@ -258,11 +260,12 @@ class HeapQuickSelectSketch extends HeapUpdateSketch {
//must rebuild or resize
if (lgArrLongs_ <= lgNomLongs_) { //resize
resizeCache();
+ return InsertedCountIncrementedResized;
}
- else { //Already at tgt size, must rebuild
- assert (lgArrLongs_ == (lgNomLongs_ + 1)) : "lgArr: " + lgArrLongs_ + ", lgNom: " + lgNomLongs_;
- quickSelectAndRebuild(); //Changes thetaLong_, curCount_, reassigns cache
- }
+ //Already at tgt size, must rebuild
+ assert (lgArrLongs_ == (lgNomLongs_ + 1)) : "lgArr: " + lgArrLongs_ + ", lgNom: " + lgNomLongs_;
+ quickSelectAndRebuild(); //Changes thetaLong_, curCount_, reassigns cache
+ return InsertedCountIncrementedRebuilt;
}
return InsertedCountIncremented;
}
diff --git a/src/main/java/org/apache/datasketches/theta/UpdateReturnState.java b/src/main/java/org/apache/datasketches/theta/UpdateReturnState.java
index df95c9d..a6d5348 100644
--- a/src/main/java/org/apache/datasketches/theta/UpdateReturnState.java
+++ b/src/main/java/org/apache/datasketches/theta/UpdateReturnState.java
@@ -27,39 +27,51 @@ package org.apache.datasketches.theta;
public enum UpdateReturnState {
/**
- * Indicates that the value was accepted into the sketch and the retained count was incremented.
+ * The hash was accepted into the sketch and the retained count was incremented.
*/
InsertedCountIncremented, //all UpdateSketches
/**
- * Indicates that the value was accepted into the sketch and the retained count was not incremented.
+ * The hash was accepted into the sketch, the retained count was incremented.
+ * The current cache was out of room and resized larger based on the Resize Factor.
+ */
+ InsertedCountIncrementedResized, //used by HeapQuickSelectSketch
+
+ /**
+ * The hash was accepted into the sketch, the retained count was incremented.
+ * The current cache was out of room and at maximum size, so the cache was rebuilt.
+ */
+ InsertedCountIncrementedRebuilt, //used by HeapQuickSelectSketch
+
+ /**
+ * The hash was accepted into the sketch and the retained count was not incremented.
*/
InsertedCountNotIncremented, //used by enhancedHashInsert for Alpha
/**
- * Indicates that the value was inserted into the local concurrent buffer,
+ * The hash was inserted into the local concurrent buffer,
* but has not yet been propagated to the concurrent shared sketch.
*/
ConcurrentBufferInserted, //used by ConcurrentHeapThetaBuffer
/**
- * Indicates that the value has been propagated to the concurrent shared sketch.
+ * The hash has been propagated to the concurrent shared sketch.
* This does not reflect the action taken by the shared sketch.
*/
ConcurrentPropagated, //used by ConcurrentHeapThetaBuffer
/**
- * Indicates that the value was rejected as a duplicate.
+ * The hash was rejected as a duplicate.
*/
RejectedDuplicate, //all UpdateSketches hashUpdate(), enhancedHashInsert
/**
- * Indicates that the value was rejected because it was null or empty.
+ * The hash was rejected because it was null or empty.
*/
RejectedNullOrEmpty, //UpdateSketch.update(arr[])
/**
- * Indicates that the value was rejected because the hash value was negative, zero or
+ * The hash was rejected because the value was negative, zero or
* greater than theta.
*/
RejectedOverTheta; //all UpdateSketches.hashUpdate()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[incubator-datasketches-java] 05/10: Deprecated
getCurrentBytes(boolean). Replaced by getCompactBytes() and
getCurrentBytes().
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leerho pushed a commit to branch Refactor_Theta_Tuple
in repository https://gitbox.apache.org/repos/asf/incubator-datasketches-java.git
commit 50f49bfc6f3442e3d92b93436242fb01e1b3bbe3
Author: Lee Rhodes <le...@users.noreply.github.com>
AuthorDate: Mon Jun 29 10:38:08 2020 -0700
Deprecated getCurrentBytes(boolean). Replaced by getCompactBytes() and
getCurrentBytes().
---
.../theta/ConcurrentHeapThetaBuffer.java | 5 +++++
.../java/org/apache/datasketches/theta/Sketch.java | 20 +++++++++++++-------
.../org/apache/datasketches/theta/UpdateSketch.java | 4 ++--
.../apache/datasketches/theta/CompactSketchTest.java | 2 --
.../theta/ConcurrentDirectQuickSelectSketchTest.java | 2 +-
.../theta/ConcurrentHeapQuickSelectSketchTest.java | 16 ++++++++--------
.../datasketches/theta/DirectIntersectionTest.java | 15 ++++++---------
.../theta/DirectQuickSelectSketchTest.java | 6 +++---
.../apache/datasketches/theta/DirectUnionTest.java | 19 +++++++++----------
.../datasketches/theta/HeapAlphaSketchTest.java | 16 ++++++++--------
.../datasketches/theta/HeapIntersectionTest.java | 17 +++++++----------
.../theta/HeapQuickSelectSketchTest.java | 18 +++++++++---------
.../org/apache/datasketches/theta/HeapUnionTest.java | 19 +++++++++----------
.../datasketches/theta/SingleItemSketchTest.java | 2 +-
.../org/apache/datasketches/theta/SketchTest.java | 16 ++++++++--------
.../apache/datasketches/theta/UpdateSketchTest.java | 13 ++++++-------
16 files changed, 95 insertions(+), 95 deletions(-)
diff --git a/src/main/java/org/apache/datasketches/theta/ConcurrentHeapThetaBuffer.java b/src/main/java/org/apache/datasketches/theta/ConcurrentHeapThetaBuffer.java
index 64ce463..e4b18f2 100644
--- a/src/main/java/org/apache/datasketches/theta/ConcurrentHeapThetaBuffer.java
+++ b/src/main/java/org/apache/datasketches/theta/ConcurrentHeapThetaBuffer.java
@@ -78,6 +78,11 @@ final class ConcurrentHeapThetaBuffer extends HeapQuickSelectSketch {
//Sketch overrides
@Override
+ public int getCompactBytes() {
+ return shared.getCompactBytes();
+ }
+
+ @Override
public int getCurrentBytes() {
return shared.getCurrentBytes();
}
diff --git a/src/main/java/org/apache/datasketches/theta/Sketch.java b/src/main/java/org/apache/datasketches/theta/Sketch.java
index eac37d0..aea0657 100644
--- a/src/main/java/org/apache/datasketches/theta/Sketch.java
+++ b/src/main/java/org/apache/datasketches/theta/Sketch.java
@@ -97,7 +97,7 @@ public abstract class Sketch {
* Wrap takes the sketch image in Memory and refers to it directly. There is no data copying onto
* the java heap. Only "Direct" Serialization Version 3 (i.e, OpenSource) sketches that have
* been explicitly stored as direct objects can be wrapped. This method assumes the
- * {@link Util#DEFAULT_UPDATE_SEED}.
+ * {@link org.apache.datasketches.Util#DEFAULT_UPDATE_SEED}.
* <a href="{@docRoot}/resources/dictionary.html#defaultUpdateSeed">Default Update Seed</a>.
* @param srcMem an image of a Sketch where the image seed hash matches the default seed hash.
* <a href="{@docRoot}/resources/dictionary.html#mem">See Memory</a>
@@ -250,6 +250,18 @@ public abstract class Sketch {
/**
* Returns the number of storage bytes required for this Sketch in its current state.
+ * @param compact if true, returns the bytes required for compact form.
+ * If this sketch is already in compact form this parameter is ignored.
+ * @return the number of storage bytes required for this sketch
+ * @deprecated use either {@link #getCompactBytes()} or {@link #getCurrentBytes()}.
+ */
+ @Deprecated
+ public int getCurrentBytes(final boolean compact) {
+ return compact ? getCompactBytes() : getCurrentBytes();
+ }
+
+ /**
+ * Returns the number of storage bytes required for this sketch in its current state.
*
* @return the number of storage bytes required for this sketch
*/
@@ -563,12 +575,6 @@ public abstract class Sketch {
abstract int getCurrentDataLongs();
- int getCurrentDataLongs(final boolean compact) {
- return (isCompact() || compact)
- ? getRetainedEntries(true)
- : (1 << ((UpdateSketch) this).getLgArrLongs());
- }
-
/**
* Returns preamble longs if stored in current state.
* @param compact if true, returns the preamble longs required for compact form.
diff --git a/src/main/java/org/apache/datasketches/theta/UpdateSketch.java b/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
index de519bf..bed93a5 100644
--- a/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
@@ -65,7 +65,7 @@ public abstract class UpdateSketch extends Sketch {
* Wrap takes the sketch image in Memory and refers to it directly. There is no data copying onto
* the java heap. Only "Direct" Serialization Version 3 (i.e, OpenSource) sketches that have
* been explicitly stored as direct objects can be wrapped. This method assumes the
- * {@link Util#DEFAULT_UPDATE_SEED}.
+ * {@link org.apache.datasketches.Util#DEFAULT_UPDATE_SEED}.
* <a href="{@docRoot}/resources/dictionary.html#defaultUpdateSeed">Default Update Seed</a>.
* @param srcMem an image of a Sketch where the image seed hash matches the default seed hash.
* <a href="{@docRoot}/resources/dictionary.html#mem">See Memory</a>
@@ -106,7 +106,7 @@ public abstract class UpdateSketch extends Sketch {
/**
* Instantiates an on-heap UpdateSketch from Memory. This method assumes the
- * {@link Util#DEFAULT_UPDATE_SEED}.
+ * {@link org.apache.datasketches.Util#DEFAULT_UPDATE_SEED}.
* @param srcMem <a href="{@docRoot}/resources/dictionary.html#mem">See Memory</a>
* @return an UpdateSketch
*/
diff --git a/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java b/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
index a2807cb..502dd4b 100644
--- a/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
@@ -185,7 +185,6 @@ public class CompactSketchTest {
public void checkMemTooSmall() {
int k = 512;
int u = k;
- boolean compact = true;
boolean ordered = false;
UpdateSketch usk = UpdateSketch.builder().setNominalEntries(k).build();
for (int i=0; i<u; i++) {
@@ -202,7 +201,6 @@ public class CompactSketchTest {
public void checkMemTooSmallOrdered() {
int k = 512;
int u = k;
- boolean compact = true;
boolean ordered = true;
UpdateSketch usk = UpdateSketch.builder().setNominalEntries(k).build();
for (int i=0; i<u; i++) {
diff --git a/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java
index 5bb93e8..4a8e836 100644
--- a/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java
@@ -382,7 +382,7 @@ public class ConcurrentDirectQuickSelectSketchTest {
assertEquals(csk.isEstimationMode(), estimating);
assertEquals(csk.getClass().getSimpleName(), "HeapCompactSketch");
- int bytes = local.getCurrentBytes(); //TODO WHAT IS GOING ON HERE
+ int bytes = local.getCompactBytes(); //TODO WHAT IS GOING ON HERE
assertEquals(bytes, (k*8) + (Family.COMPACT.getMaxPreLongs() << 3));
byte[] memArr2 = new byte[bytes];
WritableMemory mem2 = WritableMemory.wrap(memArr2);
diff --git a/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java
index 61bd5be..e119dc5 100644
--- a/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java
@@ -296,8 +296,8 @@ public class ConcurrentHeapQuickSelectSketchTest {
double localEst = local.getEstimate();
double localLB = local.getLowerBound(2);
double localUB = local.getUpperBound(2);
- int localBytes = local.getCurrentBytes(false); //size stored as UpdateSketch
- int localCompBytes = local.getCurrentBytes(true); //size stored as CompactSketch
+ int localBytes = local.getCurrentBytes(); //size stored as UpdateSketch
+ int localCompBytes = local.getCompactBytes(); //size stored as CompactSketch
assertEquals(localBytes, maxBytes);
assertEquals(local.isEstimationMode(), estimating);
@@ -310,7 +310,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(comp1.getUpperBound(2), localUB);
assertEquals(comp1.isEmpty(), false);
assertEquals(comp1.isEstimationMode(), estimating);
- assertEquals(comp1.getCurrentBytes(true), localCompBytes);
+ assertEquals(comp1.getCompactBytes(), localCompBytes);
assertEquals(comp1.getClass().getSimpleName(), "HeapCompactSketch");
comp2 = shared.compact(true, null);
@@ -320,7 +320,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(comp2.getUpperBound(2), localUB);
assertEquals(comp2.isEmpty(), false);
assertEquals(comp2.isEstimationMode(), estimating);
- assertEquals(comp2.getCurrentBytes(true), localCompBytes);
+ assertEquals(comp2.getCompactBytes(), localCompBytes);
assertEquals(comp2.getClass().getSimpleName(), "HeapCompactSketch");
byte[] memArr2 = new byte[localCompBytes];
@@ -333,7 +333,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(comp3.getUpperBound(2), localUB);
assertEquals(comp3.isEmpty(), false);
assertEquals(comp3.isEstimationMode(), estimating);
- assertEquals(comp3.getCurrentBytes(true), localCompBytes);
+ assertEquals(comp3.getCompactBytes(), localCompBytes);
assertEquals(comp3.getClass().getSimpleName(), "DirectCompactSketch");
mem2.clear();
@@ -344,7 +344,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(comp4.getUpperBound(2), localUB);
assertEquals(comp4.isEmpty(), false);
assertEquals(comp4.isEstimationMode(), estimating);
- assertEquals(comp4.getCurrentBytes(true), localCompBytes);
+ assertEquals(comp4.getCompactBytes(), localCompBytes);
assertEquals(comp4.getClass().getSimpleName(), "DirectCompactSketch");
comp4.toString(false, true, 0, false);
}
@@ -368,7 +368,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
double localUB = local.getUpperBound(2);
//int currentUSBytes = local.getCurrentBytes(false);
//assertEquals(currentUSBytes, (32*8) + 24); // clumsy, but a function of RF and TCF
- int compBytes = local.getCurrentBytes(true); //compact form
+ int compBytes = local.getCompactBytes(); //compact form
assertEquals(compBytes, 8);
assertEquals(local.isEstimationMode(), estimating);
@@ -582,7 +582,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
double uskUB = local.getUpperBound(2);
assertFalse(local.isEstimationMode());
- int bytes = local.getCurrentBytes(true); //compact form
+ int bytes = local.getCompactBytes(); //compact form
assertEquals(bytes, 8);
byte[] memArr2 = new byte[bytes];
WritableMemory mem2 = WritableMemory.wrap(memArr2);
diff --git a/src/test/java/org/apache/datasketches/theta/DirectIntersectionTest.java b/src/test/java/org/apache/datasketches/theta/DirectIntersectionTest.java
index 2b2b76e..e794ac6 100644
--- a/src/test/java/org/apache/datasketches/theta/DirectIntersectionTest.java
+++ b/src/test/java/org/apache/datasketches/theta/DirectIntersectionTest.java
@@ -29,14 +29,13 @@ import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertTrue;
import static org.testng.Assert.fail;
-import org.testng.annotations.Test;
-
-import org.apache.datasketches.memory.Memory;
-import org.apache.datasketches.memory.WritableMemory;
import org.apache.datasketches.Family;
import org.apache.datasketches.SketchesArgumentException;
import org.apache.datasketches.SketchesReadOnlyException;
import org.apache.datasketches.SketchesStateException;
+import org.apache.datasketches.memory.Memory;
+import org.apache.datasketches.memory.WritableMemory;
+import org.testng.annotations.Test;
/**
* @author Lee Rhodes
@@ -83,8 +82,7 @@ public class DirectIntersectionTest {
rsk1 = inter.getResult(ordered, null);
assertEquals(rsk1.getEstimate(), 0.0);
- boolean compact = true;
- int bytes = rsk1.getCurrentBytes(compact);
+ int bytes = rsk1.getCompactBytes();
byte[] byteArray = new byte[bytes];
WritableMemory mem = WritableMemory.wrap(byteArray);
@@ -128,8 +126,7 @@ public class DirectIntersectionTest {
rsk1 = inter.getResult(ordered, null);
assertEquals(rsk1.getEstimate(), (double)k);
- boolean compact = true;
- int bytes = rsk1.getCurrentBytes(compact);
+ int bytes = rsk1.getCompactBytes();
byte[] byteArray = new byte[bytes];
WritableMemory mem = WritableMemory.wrap(byteArray);
@@ -720,7 +717,7 @@ public class DirectIntersectionTest {
Intersection inter = Sketches.setOperationBuilder().buildIntersection(iMem);
inter.update(sk);
CompactSketch csk = inter.getResult();
- assertEquals(csk.getCurrentBytes(true), 8);
+ assertEquals(csk.getCompactBytes(), 8);
}
@Test
diff --git a/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java
index d44625d..0eaf1e4 100644
--- a/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java
@@ -218,7 +218,7 @@ public class DirectQuickSelectSketchTest {
for (int i=0; i< k; i++) { usk.update(i); }
- int bytes = usk.getCurrentBytes(false);
+ int bytes = usk.getCurrentBytes();
byte[] byteArray = usk.toByteArray();
assertEquals(bytes, byteArray.length);
@@ -336,7 +336,7 @@ public class DirectQuickSelectSketchTest {
assertEquals(csk.isEstimationMode(), estimating);
assertEquals(csk.getClass().getSimpleName(), "HeapCompactSketch");
- int bytes = usk.getCurrentBytes(true);
+ int bytes = usk.getCompactBytes();
assertEquals(bytes, (k*8) + (Family.COMPACT.getMaxPreLongs() << 3));
byte[] memArr2 = new byte[bytes];
WritableMemory mem2 = WritableMemory.wrap(memArr2);
@@ -377,7 +377,7 @@ public class DirectQuickSelectSketchTest {
double uskUB = usk.getUpperBound(2);
assertEquals(usk.isEstimationMode(), false);
- int bytes = usk.getCurrentBytes(true); //compact form
+ int bytes = usk.getCompactBytes(); //compact form
assertEquals(bytes, 8);
byte[] memArr2 = new byte[bytes];
WritableMemory mem2 = WritableMemory.wrap(memArr2);
diff --git a/src/test/java/org/apache/datasketches/theta/DirectUnionTest.java b/src/test/java/org/apache/datasketches/theta/DirectUnionTest.java
index 840d2de..0975b40 100644
--- a/src/test/java/org/apache/datasketches/theta/DirectUnionTest.java
+++ b/src/test/java/org/apache/datasketches/theta/DirectUnionTest.java
@@ -19,25 +19,24 @@
package org.apache.datasketches.theta;
+import static java.nio.charset.StandardCharsets.UTF_8;
import static org.apache.datasketches.theta.BackwardConversions.convertSerVer3toSerVer1;
import static org.apache.datasketches.theta.BackwardConversions.convertSerVer3toSerVer2;
import static org.apache.datasketches.theta.HeapUnionTest.testAllCompactForms;
import static org.apache.datasketches.theta.PreambleUtil.SER_VER_BYTE;
import static org.apache.datasketches.theta.SetOperation.getMaxUnionBytes;
-import static java.nio.charset.StandardCharsets.UTF_8;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertTrue;
import java.util.Arrays;
-import org.testng.annotations.Test;
-
-import org.apache.datasketches.memory.Memory;
-import org.apache.datasketches.memory.WritableMemory;
import org.apache.datasketches.Family;
import org.apache.datasketches.SketchesArgumentException;
import org.apache.datasketches.Util;
+import org.apache.datasketches.memory.Memory;
+import org.apache.datasketches.memory.WritableMemory;
+import org.testng.annotations.Test;
/**
* @author Lee Rhodes
@@ -271,7 +270,7 @@ public class DirectUnionTest {
usk2.update(i); //2k no overlap, exact, will force early stop
}
- WritableMemory cskMem2 = WritableMemory.wrap(new byte[usk2.getCurrentBytes(true)]);
+ WritableMemory cskMem2 = WritableMemory.wrap(new byte[usk2.getCompactBytes()]);
CompactSketch cosk2 = usk2.compact(true, cskMem2); //ordered, loads the cskMem2 as ordered
WritableMemory uMem = WritableMemory.wrap(new byte[getMaxUnionBytes(k)]); //union memory
@@ -311,7 +310,7 @@ public class DirectUnionTest {
usk2.update(i); //2k no overlap, exact, will force early stop
}
- WritableMemory cskMem2 = WritableMemory.wrap(new byte[usk2.getCurrentBytes(true)]);
+ WritableMemory cskMem2 = WritableMemory.wrap(new byte[usk2.getCompactBytes()]);
usk2.compact(true, cskMem2); //ordered, loads the cskMem2 as ordered
WritableMemory uMem = WritableMemory.wrap(new byte[getMaxUnionBytes(k)]); //union memory
@@ -351,7 +350,7 @@ public class DirectUnionTest {
usk2.update(i); //2k no overlap, exact, will force early stop
}
- WritableMemory cskMem2 = WritableMemory.wrap(new byte[usk2.getCurrentBytes(true)]);
+ WritableMemory cskMem2 = WritableMemory.wrap(new byte[usk2.getCompactBytes()]);
usk2.compact(false, cskMem2); //unordered, loads the cskMem2 as unordered
WritableMemory uMem = WritableMemory.wrap(new byte[getMaxUnionBytes(k)]); //union memory
@@ -683,7 +682,7 @@ public class DirectUnionTest {
Union union = Sketches.setOperationBuilder().setNominalEntries(k).buildUnion(iMem);
union.update(sk);
CompactSketch csk = union.getResult();
- assertEquals(csk.getCurrentBytes(true), 8);
+ assertEquals(csk.getCompactBytes(), 8);
}
@Test
@@ -795,7 +794,7 @@ public class DirectUnionTest {
usk.update(Integer.toString(i));
}
usk.rebuild(); //optional but created the symptom
- final WritableMemory memIn = WritableMemory.allocate(usk.getCurrentBytes(true));
+ final WritableMemory memIn = WritableMemory.allocate(usk.getCompactBytes());
usk.compact(true, memIn); //side effect of loading the memIn
//create empty target union in off-heap mem
diff --git a/src/test/java/org/apache/datasketches/theta/HeapAlphaSketchTest.java b/src/test/java/org/apache/datasketches/theta/HeapAlphaSketchTest.java
index 62ff8ef..c77ff8d 100644
--- a/src/test/java/org/apache/datasketches/theta/HeapAlphaSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/HeapAlphaSketchTest.java
@@ -140,7 +140,7 @@ public class HeapAlphaSketchTest {
usk.update(i);
}
- int bytes = usk.getCurrentBytes(false);
+ int bytes = usk.getCurrentBytes();
byte[] byteArray = usk.toByteArray();
assertEquals(bytes, byteArray.length);
@@ -245,8 +245,8 @@ public class HeapAlphaSketchTest {
double comp1est = comp1.getEstimate();
double comp1lb = comp1.getLowerBound(2);
double comp1ub = comp1.getUpperBound(2);
- int comp1bytes = comp1.getCurrentBytes(true);
- assertEquals(comp1bytes, comp1.getCurrentBytes(false));
+ int comp1bytes = comp1.getCompactBytes();
+ assertEquals(comp1bytes, comp1.getCurrentBytes());
int comp1curCount = comp1.getRetainedEntries(true); //flag is not relevant
assertEquals(comp1bytes, (comp1curCount << 3) + (Family.COMPACT.getMaxPreLongs() << 3));
@@ -261,11 +261,11 @@ public class HeapAlphaSketchTest {
assertEquals(comp2.getUpperBound(2), comp1ub);
assertEquals(comp2.isEmpty(), false);
assertEquals(comp2.isEstimationMode(), estimating);
- assertEquals(comp1bytes, comp2.getCurrentBytes(true)); //flag is not relevant
+ assertEquals(comp1bytes, comp2.getCompactBytes()); //flag is not relevant
assertEquals(comp1curCount, comp2.getRetainedEntries(true)); //flag is not relevant
assertEquals(comp2.getClass().getSimpleName(), "HeapCompactSketch");
- int bytes = usk.getCurrentBytes(true);
+ int bytes = usk.getCompactBytes();
int alphaBytes = sk1.getRetainedEntries(true) * 8;
assertEquals(bytes, alphaBytes + (Family.COMPACT.getMaxPreLongs() << 3));
byte[] memArr2 = new byte[bytes];
@@ -278,7 +278,7 @@ public class HeapAlphaSketchTest {
assertEquals(comp3.getUpperBound(2), comp1ub);
assertEquals(comp3.isEmpty(), false);
assertEquals(comp3.isEstimationMode(), estimating);
- assertEquals(comp1bytes, comp3.getCurrentBytes(true)); //flag is not relevant
+ assertEquals(comp1bytes, comp3.getCompactBytes()); //flag is not relevant
assertEquals(comp1curCount, comp3.getRetainedEntries(true)); //flag is not relevant
assertEquals(comp3.getClass().getSimpleName(), "DirectCompactSketch");
@@ -290,7 +290,7 @@ public class HeapAlphaSketchTest {
assertEquals(comp4.getUpperBound(2), comp1ub);
assertEquals(comp4.isEmpty(), false);
assertEquals(comp4.isEstimationMode(), estimating);
- assertEquals(comp1bytes, comp4.getCurrentBytes(true)); //flag is not relevant
+ assertEquals(comp1bytes, comp4.getCompactBytes()); //flag is not relevant
assertEquals(comp1curCount, comp4.getRetainedEntries(true)); //flag is not relevant
assertEquals(comp4.getClass().getSimpleName(), "DirectCompactSketch");
}
@@ -310,7 +310,7 @@ public class HeapAlphaSketchTest {
double uskUB = usk.getUpperBound(2);
assertEquals(usk.isEstimationMode(), estimating);
- int bytes = usk.getCurrentBytes(true);
+ int bytes = usk.getCompactBytes();
assertEquals(bytes, 8); //compact, empty and theta = 1.0
byte[] memArr2 = new byte[bytes];
WritableMemory mem2 = WritableMemory.wrap(memArr2);
diff --git a/src/test/java/org/apache/datasketches/theta/HeapIntersectionTest.java b/src/test/java/org/apache/datasketches/theta/HeapIntersectionTest.java
index cdb1f9e..4008203 100644
--- a/src/test/java/org/apache/datasketches/theta/HeapIntersectionTest.java
+++ b/src/test/java/org/apache/datasketches/theta/HeapIntersectionTest.java
@@ -27,13 +27,12 @@ import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertTrue;
-import org.testng.annotations.Test;
-
-import org.apache.datasketches.memory.Memory;
-import org.apache.datasketches.memory.WritableMemory;
import org.apache.datasketches.Family;
import org.apache.datasketches.SketchesArgumentException;
import org.apache.datasketches.SketchesStateException;
+import org.apache.datasketches.memory.Memory;
+import org.apache.datasketches.memory.WritableMemory;
+import org.testng.annotations.Test;
/**
* @author Lee Rhodes
@@ -71,8 +70,7 @@ public class HeapIntersectionTest {
rsk1 = inter.getResult(ordered, null);
assertEquals(rsk1.getEstimate(), 0.0);
- boolean compact = true;
- int bytes = rsk1.getCurrentBytes(compact);
+ int bytes = rsk1.getCompactBytes();
byte[] byteArray = new byte[bytes];
WritableMemory mem = WritableMemory.wrap(byteArray);
@@ -114,8 +112,7 @@ public class HeapIntersectionTest {
rsk1 = inter.getResult(ordered, null);
assertEquals(rsk1.getEstimate(), (double)k);
- boolean compact = true;
- int bytes = rsk1.getCurrentBytes(compact);
+ int bytes = rsk1.getCompactBytes();
byte[] byteArray = new byte[bytes];
WritableMemory mem = WritableMemory.wrap(byteArray);
@@ -561,7 +558,7 @@ public class HeapIntersectionTest {
Intersection inter = Sketches.setOperationBuilder().buildIntersection();
inter.update(sk);
CompactSketch csk = inter.getResult();
- assertEquals(csk.getCurrentBytes(true), 8);
+ assertEquals(csk.getCompactBytes(), 8);
}
@Test
@@ -576,7 +573,7 @@ public class HeapIntersectionTest {
UpdateSketch skB = Sketches.updateSketchBuilder().build();
Intersection inter = Sketches.setOperationBuilder().buildIntersection();
CompactSketch csk = inter.intersect(skA, skB);
- assertEquals(csk.getCurrentBytes(true), 8);
+ assertEquals(csk.getCompactBytes(), 8);
}
@Test
diff --git a/src/test/java/org/apache/datasketches/theta/HeapQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/HeapQuickSelectSketchTest.java
index c230a88..74a312f 100644
--- a/src/test/java/org/apache/datasketches/theta/HeapQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/HeapQuickSelectSketchTest.java
@@ -133,7 +133,7 @@ public class HeapQuickSelectSketchTest {
usk.update(i);
}
- int bytes = usk.getCurrentBytes(false);
+ int bytes = usk.getCurrentBytes();
byte[] byteArray = usk.toByteArray();
assertEquals(bytes, byteArray.length);
@@ -236,8 +236,8 @@ public class HeapQuickSelectSketchTest {
double uskEst = usk.getEstimate();
double uskLB = usk.getLowerBound(2);
double uskUB = usk.getUpperBound(2);
- int uskBytes = usk.getCurrentBytes(false); //size stored as UpdateSketch
- int uskCompBytes = usk.getCurrentBytes(true); //size stored as CompactSketch
+ int uskBytes = usk.getCurrentBytes(); //size stored as UpdateSketch
+ int uskCompBytes = usk.getCompactBytes(); //size stored as CompactSketch
assertEquals(uskBytes, maxBytes);
assertEquals(usk.isEstimationMode(), estimating);
@@ -250,7 +250,7 @@ public class HeapQuickSelectSketchTest {
assertEquals(comp1.getUpperBound(2), uskUB);
assertEquals(comp1.isEmpty(), false);
assertEquals(comp1.isEstimationMode(), estimating);
- assertEquals(comp1.getCurrentBytes(true), uskCompBytes);
+ assertEquals(comp1.getCompactBytes(), uskCompBytes);
assertEquals(comp1.getClass().getSimpleName(), "HeapCompactSketch");
comp2 = usk.compact(true, null);
@@ -260,7 +260,7 @@ public class HeapQuickSelectSketchTest {
assertEquals(comp2.getUpperBound(2), uskUB);
assertEquals(comp2.isEmpty(), false);
assertEquals(comp2.isEstimationMode(), estimating);
- assertEquals(comp2.getCurrentBytes(true), uskCompBytes);
+ assertEquals(comp2.getCompactBytes(), uskCompBytes);
assertEquals(comp2.getClass().getSimpleName(), "HeapCompactSketch");
byte[] memArr2 = new byte[uskCompBytes];
@@ -273,7 +273,7 @@ public class HeapQuickSelectSketchTest {
assertEquals(comp3.getUpperBound(2), uskUB);
assertEquals(comp3.isEmpty(), false);
assertEquals(comp3.isEstimationMode(), estimating);
- assertEquals(comp3.getCurrentBytes(true), uskCompBytes);
+ assertEquals(comp3.getCompactBytes(), uskCompBytes);
assertEquals(comp3.getClass().getSimpleName(), "DirectCompactSketch");
mem2.clear();
@@ -284,7 +284,7 @@ public class HeapQuickSelectSketchTest {
assertEquals(comp4.getUpperBound(2), uskUB);
assertEquals(comp4.isEmpty(), false);
assertEquals(comp4.isEstimationMode(), estimating);
- assertEquals(comp4.getCurrentBytes(true), uskCompBytes);
+ assertEquals(comp4.getCompactBytes(), uskCompBytes);
assertEquals(comp4.getClass().getSimpleName(), "DirectCompactSketch");
comp4.toString(false, true, 0, false);
}
@@ -303,9 +303,9 @@ public class HeapQuickSelectSketchTest {
double uskEst = usk.getEstimate();
double uskLB = usk.getLowerBound(2);
double uskUB = usk.getUpperBound(2);
- int currentUSBytes = usk.getCurrentBytes(false);
+ int currentUSBytes = usk.getCurrentBytes();
assertEquals(currentUSBytes, (32*8) + 24); // clumsy, but a function of RF and TCF
- int compBytes = usk.getCurrentBytes(true); //compact form
+ int compBytes = usk.getCompactBytes(); //compact form
assertEquals(compBytes, 8);
assertEquals(usk.isEstimationMode(), estimating);
diff --git a/src/test/java/org/apache/datasketches/theta/HeapUnionTest.java b/src/test/java/org/apache/datasketches/theta/HeapUnionTest.java
index fcacb03..138112e 100644
--- a/src/test/java/org/apache/datasketches/theta/HeapUnionTest.java
+++ b/src/test/java/org/apache/datasketches/theta/HeapUnionTest.java
@@ -19,22 +19,21 @@
package org.apache.datasketches.theta;
+import static java.nio.charset.StandardCharsets.UTF_8;
import static org.apache.datasketches.theta.BackwardConversions.convertSerVer3toSerVer1;
import static org.apache.datasketches.theta.BackwardConversions.convertSerVer3toSerVer2;
import static org.apache.datasketches.theta.PreambleUtil.SER_VER_BYTE;
-import static java.nio.charset.StandardCharsets.UTF_8;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertFalse;
import java.util.Arrays;
-import org.testng.annotations.Test;
-
-import org.apache.datasketches.memory.Memory;
-import org.apache.datasketches.memory.WritableMemory;
import org.apache.datasketches.Family;
import org.apache.datasketches.SketchesArgumentException;
import org.apache.datasketches.Util;
+import org.apache.datasketches.memory.Memory;
+import org.apache.datasketches.memory.WritableMemory;
+import org.testng.annotations.Test;
/**
* @author Lee Rhodes
@@ -230,7 +229,7 @@ public class HeapUnionTest {
usk2.update(i); //2k no overlap, exact, will force early stop
}
- WritableMemory cskMem2 = WritableMemory.wrap(new byte[usk2.getCurrentBytes(true)]);
+ WritableMemory cskMem2 = WritableMemory.wrap(new byte[usk2.getCompactBytes()]);
CompactSketch cosk2 = usk2.compact(true, cskMem2); //ordered, loads the cskMem2 as ordered
Union union = SetOperation.builder().setNominalEntries(k).buildUnion();
@@ -269,7 +268,7 @@ public class HeapUnionTest {
usk2.update(i); //2k no overlap, exact, will force early stop
}
- WritableMemory cskMem2 = WritableMemory.wrap(new byte[usk2.getCurrentBytes(true)]);
+ WritableMemory cskMem2 = WritableMemory.wrap(new byte[usk2.getCompactBytes()]);
usk2.compact(true, cskMem2); //ordered, loads the cskMem2 as ordered
Union union = SetOperation.builder().setNominalEntries(k).buildUnion();
@@ -308,7 +307,7 @@ public class HeapUnionTest {
usk2.update(i); //2k no overlap, exact, will force early stop
}
- WritableMemory cskMem2 = WritableMemory.wrap(new byte[usk2.getCurrentBytes(true)]);
+ WritableMemory cskMem2 = WritableMemory.wrap(new byte[usk2.getCompactBytes()]);
usk2.compact(false, cskMem2); //unordered, loads the cskMem2 as unordered
Union union = SetOperation.builder().setNominalEntries(k).buildUnion();
@@ -562,7 +561,7 @@ public class HeapUnionTest {
Union union = Sketches.setOperationBuilder().setNominalEntries(k).buildUnion();
union.update(sk);
CompactSketch csk = union.getResult();
- assertEquals(csk.getCurrentBytes(true), 8);
+ assertEquals(csk.getCompactBytes(), 8);
}
@Test
@@ -626,7 +625,7 @@ public class HeapUnionTest {
compEst2 = comp2.getEstimate();
assertEquals(compEst2, compEst1, 0.0);
- WritableMemory mem = WritableMemory.wrap(new byte[comp2.getCurrentBytes(false)]);
+ WritableMemory mem = WritableMemory.wrap(new byte[comp2.getCurrentBytes()]);
compEst2 = union.getResult(false, mem).getEstimate(); //not ordered, mem
assertEquals(compEst2, compEst1, 0.0);
diff --git a/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java b/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java
index dbbc6ff..d2fa4db 100644
--- a/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java
@@ -122,7 +122,7 @@ public class SingleItemSketchTest {
@Test
public void checkSketchInterface() {
SingleItemSketch sis = SingleItemSketch.create(1);
- assertEquals(sis.getCurrentBytes(true), 16);
+ assertEquals(sis.getCompactBytes(), 16);
assertEquals(sis.getEstimate(), 1.0);
assertEquals(sis.getLowerBound(1), 1.0);
assertEquals(sis.getRetainedEntries(true), 1);
diff --git a/src/test/java/org/apache/datasketches/theta/SketchTest.java b/src/test/java/org/apache/datasketches/theta/SketchTest.java
index 840b31e..fa11112 100644
--- a/src/test/java/org/apache/datasketches/theta/SketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/SketchTest.java
@@ -68,12 +68,12 @@ public class SketchTest {
assertEquals(sketch.getCurrentPreambleLongs(false), lowQSPreLongs);
assertEquals(sketch.getCurrentPreambleLongs(true), 1); //compact form
assertEquals(sketch.getCurrentDataLongs(), k*2);
- assertEquals(sketch.getCurrentBytes(false), (k*2*8) + (lowQSPreLongs << 3));
- assertEquals(sketch.getCurrentBytes(true), lowCompPreLongs << 3);
+ assertEquals(sketch.getCurrentBytes(), (k*2*8) + (lowQSPreLongs << 3));
+ assertEquals(sketch.getCompactBytes(), lowCompPreLongs << 3);
CompactSketch compSk = sketch.compact(false, null);
- assertEquals(compSk.getCurrentBytes(true), 8);
- assertEquals(compSk.getCurrentBytes(false), 8);
+ assertEquals(compSk.getCompactBytes(), 8);
+ assertEquals(compSk.getCurrentBytes(), 8);
assertEquals(compSk.getCurrentDataLongs(), 0);
int compPreLongs = computeCompactPreLongs(sketch.isEmpty(), sketch.getRetainedEntries(true),
@@ -87,8 +87,8 @@ public class SketchTest {
assertEquals(sketch.getCurrentPreambleLongs(false), lowQSPreLongs);
assertEquals(sketch.getCurrentPreambleLongs(true), 2); //compact form
assertEquals(sketch.getCurrentDataLongs(), k*2);
- assertEquals(sketch.getCurrentBytes(false), (k*2*8) + (lowQSPreLongs << 3));
- assertEquals(sketch.getCurrentBytes(true), (k*8) + (2*8)); //compact form //FAILS HERE
+ assertEquals(sketch.getCurrentBytes(), (k*2*8) + (lowQSPreLongs << 3));
+ assertEquals(sketch.getCompactBytes(), (k*8) + (2*8)); //compact form //FAILS HERE
compPreLongs = computeCompactPreLongs(sketch.isEmpty(), sketch.getRetainedEntries(true),
sketch.getThetaLong());
@@ -102,8 +102,8 @@ public class SketchTest {
assertEquals(sketch.getCurrentPreambleLongs(false), lowQSPreLongs);
assertEquals(sketch.getCurrentPreambleLongs(true), 3); //compact form
assertEquals(sketch.getCurrentDataLongs(), k*2);
- assertEquals(sketch.getCurrentBytes(false), (k*2*8) + (lowQSPreLongs << 3));
- assertEquals(sketch.getCurrentBytes(true), (curCount*8) + (3*8)); //compact form
+ assertEquals(sketch.getCurrentBytes(), (k*2*8) + (lowQSPreLongs << 3));
+ assertEquals(sketch.getCompactBytes(), (curCount*8) + (3*8)); //compact form
compPreLongs = computeCompactPreLongs(sketch.isEmpty(), sketch.getRetainedEntries(true),
sketch.getThetaLong());
diff --git a/src/test/java/org/apache/datasketches/theta/UpdateSketchTest.java b/src/test/java/org/apache/datasketches/theta/UpdateSketchTest.java
index c5bb6c2..0b5d9da 100644
--- a/src/test/java/org/apache/datasketches/theta/UpdateSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/UpdateSketchTest.java
@@ -19,22 +19,21 @@
package org.apache.datasketches.theta;
+import static java.nio.charset.StandardCharsets.UTF_8;
import static org.apache.datasketches.Util.DEFAULT_UPDATE_SEED;
import static org.apache.datasketches.theta.PreambleUtil.PREAMBLE_LONGS_BYTE;
import static org.apache.datasketches.theta.PreambleUtil.SER_VER_BYTE;
-import static java.nio.charset.StandardCharsets.UTF_8;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.fail;
-import org.testng.annotations.Test;
-
-import org.apache.datasketches.memory.DefaultMemoryRequestServer;
-import org.apache.datasketches.memory.MemoryRequestServer;
-import org.apache.datasketches.memory.WritableMemory;
import org.apache.datasketches.Family;
import org.apache.datasketches.ResizeFactor;
import org.apache.datasketches.SketchesArgumentException;
import org.apache.datasketches.Util;
+import org.apache.datasketches.memory.DefaultMemoryRequestServer;
+import org.apache.datasketches.memory.MemoryRequestServer;
+import org.apache.datasketches.memory.WritableMemory;
+import org.testng.annotations.Test;
/**
* @author Lee Rhodes
@@ -147,7 +146,7 @@ public class UpdateSketchTest {
public void checkCompact() {
UpdateSketch sk = Sketches.updateSketchBuilder().build();
CompactSketch csk = sk.compact();
- assertEquals(csk.getCurrentBytes(true), 8);
+ assertEquals(csk.getCompactBytes(), 8);
}
@Test(expectedExceptions = SketchesArgumentException.class)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[incubator-datasketches-java] 01/10: Interim. This removed much of
the code in the PairwiseSetOperations class and replaced it with references
to the now equivalent classes that extend SetOperation.
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leerho pushed a commit to branch Refactor_Theta_Tuple
in repository https://gitbox.apache.org/repos/asf/incubator-datasketches-java.git
commit 85d44429a1b032c5ff1758c2e595b61b44613d7a
Author: Lee Rhodes <le...@users.noreply.github.com>
AuthorDate: Fri Jun 26 16:43:41 2020 -0700
Interim. This removed much of the code in the PairwiseSetOperations
class and replaced it with references to the now equivalent classes that
extend SetOperation.
---
.../java/org/apache/datasketches/theta/AnotB.java | 31 +-
.../org/apache/datasketches/theta/AnotBimpl.java | 321 ++-------------------
.../datasketches/theta/IntersectionImplR.java | 1 -
.../datasketches/theta/PairwiseSetOperations.java | 156 +---------
.../apache/datasketches/theta/SetOperation.java | 13 -
.../apache/datasketches/theta/AnotBimplTest.java | 7 +-
.../datasketches/theta/SetOpsCornerCasesTest.java | 32 +-
7 files changed, 67 insertions(+), 494 deletions(-)
diff --git a/src/main/java/org/apache/datasketches/theta/AnotB.java b/src/main/java/org/apache/datasketches/theta/AnotB.java
index 4a4527a..0c90581 100644
--- a/src/main/java/org/apache/datasketches/theta/AnotB.java
+++ b/src/main/java/org/apache/datasketches/theta/AnotB.java
@@ -24,18 +24,33 @@ import org.apache.datasketches.memory.WritableMemory;
/**
* The API for the set difference operation <i>A and not B</i> operations.
- * This is a stateless operation. However, to make the API
- * more consistent with the other set operations the intended use is:
+ * This class include both stateful and stateless operations.
+ *
+ * <p>The stateful operation is as follows:</p>
+ * <pre><code>
+ * AnotB anotb = SetOperationBuilder.buildAnotB();
+ *
+ * anotb.setA(Sketch skA); //The first argument.
+ * anotb.notB(Sketch skB); //The second (subtraction) argument.
+ * anotb.notB(Sketch skC); // ...any number of additional subtractions...
+ * anotb.getResult(false); //Get an interim result.
+ * anotb.notB(Sketch skD); //Additional subtractions.
+ * anotb.getResult(true); //Final result and resets the AnotB operator.
+ * </code></pre>
+ *
+ * <p>The stateless operation is as follows:</p>
* <pre><code>
- * AnotB aNotB = SetOperationBuilder.buildAnotB();
- * aNotB.update(SketchA, SketchB); //Called only once.
- * CompactSketch result = aNotB.getResult();
+ * AnotB anotb = SetOperationBuilder.buildAnotB();
+ *
+ * CompactSketch csk = anotb.aNotB(Sketch skA, Sketch skB);
* </code></pre>
*
- * <p>Calling the update function a second time essentially clears the internal state and updates
- * with the new pair of sketches.
+ * <p>Calling the <i>setA</i> operation a second time essentially clears the internal state and loads
+ * the new sketch.</p>
*
- * <p>As an alternative, one can use the aNotB method that returns the result immediately.
+ * <p>The stateless and stateful operations are independent of each other with the exception of
+ * sharing the same update hash seed loaded as the default seed or specified by the user as an
+ * argument to the builder.</p>
*
* @author Lee Rhodes
*/
diff --git a/src/main/java/org/apache/datasketches/theta/AnotBimpl.java b/src/main/java/org/apache/datasketches/theta/AnotBimpl.java
index 576af7b..752baac 100644
--- a/src/main/java/org/apache/datasketches/theta/AnotBimpl.java
+++ b/src/main/java/org/apache/datasketches/theta/AnotBimpl.java
@@ -19,16 +19,11 @@
package org.apache.datasketches.theta;
-import static java.lang.Math.min;
import static org.apache.datasketches.HashOperations.convertToHashTable;
-import static org.apache.datasketches.HashOperations.hashArrayInsert;
import static org.apache.datasketches.HashOperations.hashSearch;
import static org.apache.datasketches.Util.REBUILD_THRESHOLD;
import static org.apache.datasketches.Util.checkSeedHashes;
import static org.apache.datasketches.Util.simpleIntLog2;
-import static org.apache.datasketches.theta.PreambleUtil.COMPACT_FLAG_MASK;
-import static org.apache.datasketches.theta.PreambleUtil.ORDERED_FLAG_MASK;
-import static org.apache.datasketches.theta.PreambleUtil.READ_ONLY_FLAG_MASK;
import java.util.Arrays;
@@ -36,7 +31,7 @@ import org.apache.datasketches.memory.Memory;
import org.apache.datasketches.memory.WritableMemory;
/**
- * Implements the A-and-not-B operation.
+ * Implements the A-and-not-B operations.
* @author Lee Rhodes
* @author Kevin Lang
*/
@@ -47,12 +42,6 @@ final class AnotBimpl extends AnotB {
private long[] hashArr_; //compact array w curCount_ entries
private int curCount_;
- //Remove all 4 of these with deprecated
- private Sketch skA_;
- private Sketch skB_;
- private int lgArrLongsHT_; //for Hash Table only. may not need to be member after refactoring
- private long[] bHashTable_; //may not need to be member after refactoring.
-
/**
* Construct a new AnotB SetOperation on the java heap. Called by SetOperation.Builder.
*
@@ -148,55 +137,12 @@ final class AnotBimpl extends AnotB {
public CompactSketch getResult(final boolean dstOrdered, final WritableMemory dstMem,
final boolean reset) {
final CompactSketch result =
- getResult(hashArr_, empty_, seedHash_, curCount_, thetaLong_, dstOrdered, dstMem);
+ CompactOperations.componentsToCompact(
+ thetaLong_, curCount_, seedHash_, empty_, true, false, dstOrdered, dstMem, hashArr_);
if (reset) { reset(); }
return result;
}
- private static CompactSketch getResult(
- final long[] hashArr,
- final boolean empty,
- final short seedHash,
- final int curCount,
- final long thetaLong,
- final boolean dstOrdered,
- final WritableMemory dstMem) {
- final CompactSketch result;
- if (dstMem == null) { //Heap
- if (empty) { return EmptyCompactSketch.getInstance(); }
- if (curCount == 1) { return new SingleItemSketch(hashArr[0], seedHash); }
- //curCount > 1
- if (dstOrdered) {
- Arrays.sort(hashArr);
- result = new HeapCompactOrderedSketch(hashArr, false, seedHash, curCount, thetaLong);
- } else {
- result = new HeapCompactUnorderedSketch(hashArr, false, seedHash, curCount, thetaLong);
- }
- }
- else { //Direct
- if (empty) {
- dstMem.putByteArray(0, EmptyCompactSketch.EMPTY_COMPACT_SKETCH_ARR, 0, 8);
- return EmptyCompactSketch.getInstance();
- }
- if (curCount == 1) {
- final SingleItemSketch sis = new SingleItemSketch(hashArr[0], seedHash);
- dstMem.putByteArray(0, sis.toByteArray(), 0, 16);
- }
- final int preLongs = CompactOperations.computeCompactPreLongs(thetaLong, false, curCount);
- if (dstOrdered) {
- final byte flags = (byte)(READ_ONLY_FLAG_MASK | COMPACT_FLAG_MASK | ORDERED_FLAG_MASK);
- Arrays.sort(hashArr);
- CompactOperations.loadCompactMemory(hashArr, seedHash, curCount, thetaLong, dstMem, flags, preLongs);
- result = new DirectCompactOrderedSketch(dstMem);
- } else {
- final byte flags = (byte)(READ_ONLY_FLAG_MASK | COMPACT_FLAG_MASK);
- CompactOperations.loadCompactMemory(hashArr, seedHash, curCount, thetaLong, dstMem, flags, preLongs);
- result = new DirectCompactUnorderedSketch(dstMem);
- }
- }
- return result;
- }
-
@Override
public CompactSketch aNotB(final Sketch skA, final Sketch skB, final boolean dstOrdered,
final WritableMemory dstMem) {
@@ -245,7 +191,8 @@ final class AnotBimpl extends AnotB {
empty = ((nonMatches == 0) && (thetaLong == Long.MAX_VALUE));
final long[] hashArrOut = Arrays.copyOfRange(tmpHashArrA, 0, nonMatches);
final CompactSketch result =
- AnotBimpl.getResult(hashArrOut, empty, seedHash_, nonMatches, thetaLong, dstOrdered, dstMem);
+ CompactOperations.componentsToCompact(
+ thetaLong, nonMatches, seedHash_, empty, true, false, dstOrdered, dstMem, hashArrOut);
return result;
}
@@ -264,249 +211,13 @@ final class AnotBimpl extends AnotB {
return false;
}
- //Deprecated methods
-
- @Deprecated
- @Override
- public void update(final Sketch a, final Sketch b) {
- skA_ = a;
- skB_ = b;
- thetaLong_ = Long.MAX_VALUE; //*
- empty_ = true; //*
- hashArr_ = null; //*
- curCount_ = 0; //*
- lgArrLongsHT_ = 5;
- bHashTable_ = null;
- compute();
- }
-
- @Deprecated
- @Override
- public CompactSketch getResult() {
- return getResult(true, null);
- }
-
- @Deprecated
- @Override
- public CompactSketch getResult(final boolean dstOrdered, final WritableMemory dstMem) {
- final long[] compactCache = (curCount_ <= 0)
- ? new long[0]
- : Arrays.copyOfRange(hashArr_, 0, curCount_);
- if (dstOrdered && (curCount_ > 1)) {
- Arrays.sort(compactCache);
- }
- //Create the CompactSketch
- final CompactSketch csk = CompactOperations.componentsToCompact(
- thetaLong_, curCount_, seedHash_, empty_, true, dstOrdered, dstOrdered, dstMem, compactCache);
- reset();
- return csk;
- }
-
//restricted
- void compute() {
- final int swA = ((skA_ == null) || (skA_ instanceof EmptyCompactSketch))
- ? 0
- : (skA_.isEmpty())
- ? 1
- : (skA_ instanceof UpdateSketch)
- ? 4
- : (skA_.isOrdered())
- ? 3
- : 2;
- final int swB = ((skB_ == null) || (skB_ instanceof EmptyCompactSketch))
- ? 0
- : (skB_.isEmpty())
- ? 1
- : (skB_ instanceof UpdateSketch)
- ? 4
- : (skB_.isOrdered())
- ? 3
- : 2;
- final int sw = (swA * 8) | swB;
-
- // NOTES:
- // In the table below, A and B refer to the two input sketches in the order A-not-B.
- // The Theta rule: min(ThetaA, ThetaB)
- // The Empty rule: Whatever the empty state of A is: E(A)
- // The Return triple is defined as: (Theta, Count, EmptyFlag).
- // bHashTable temporarily stores the values of B.
- // A sketch in stored form can be in one of 5 states.
- // Null is not actually a state, but is included for completeness.
- // Null is interpreted as {Theta = 1.0, count = 0, empty = true}.
- // In some cases the empty state may have Theta < 1.0 but it is ignored; count must be zero.
- // State:
- // 0 N Null or instance of EmptyCompactSketch
- // 1 E Empty bit set
- // 2 C Compact, not ordered
- // 3 O Compact Ordered
- // 4 H Hash-Table
- //
- //A B swA swB Case Actions
- //N N 0 0 0 Return (1.0, 0, T)
- //N E 0 1 1 CheckB, Return (1.0, 0, T)
- //N C 0 2 2 CheckB, Return (1.0, 0, T)
- //N O 0 3 3 CheckB, Return (1.0, 0, T)
- //N H 0 4 4 CheckB, Return (1.0, 0, T)
- //E N 1 0 8 CheckA, Return (1.0, 0, T)
- //E E 1 1 9 CheckAB, Return (1.0, 0, T)
- //E C 1 2 10 CheckAB, Return (1.0, 0, T)
- //E O 1 3 11 CheckAB, Return (1.0, 0, T)
- //E H 1 4 12 CheckAB, Return (1.0, 0, T)
- //C N 2 0 16 CheckA, Return (ThA, |A|, F), copyA
- //C E 2 1 17 CheckAB, Return (ThA, |A|, F)), copyA
- //C C 2 2 18 CheckAB, B -> H; => C,H; scanAllAsearchB()
- //C O 2 3 19 CheckAB, B -> H; => C,H; scanAllAsearchB()
- //C H 2 4 20 CheckAB, scanAllAsearchB()
- //O N 3 0 24 CheckA, Return (ThA, |A|, F), copyA
- //O E 3 1 25 CheckAB, Return (ThA, |A|, F), copyA
- //O C 3 2 26 CheckAB, B -> H; => O,H; scanEarlyStopAsearchB()
- //O O 3 3 27 CheckAB, B -> H; => O,H; scanEarlyStopAsearchB()
- //O H 3 4 28 CheckAB, scanEarlyStopAsearchB()
- //H N 4 0 32 CheckA, Return (ThA, |A|, F), copyA
- //H E 4 1 33 CheckAB, Return (ThA, |A|, F), copyA
- //H C 4 2 34 CheckAB, B -> H; => H,H; scanAllAsearchB()
- //H O 4 3 35 CheckAB, B -> H; => H,H; scanAllAsearchB()
- //H H 4 4 36 CheckAB, scanAllAsearchB()
-
- switch (sw) {
- case 0 : //A Null, B Null; Return (1.0, 0, T)
- thetaLong_ = Long.MAX_VALUE;
- empty_ = true;
- break;
-
- case 10: //A Empty, B Compact; CheckAB, Return (1.0, 0, T)
- case 11: //A Empty, B Ordered; CheckAB, Return (1.0, 0, T)
- case 12: //A Empty, B HashTbl; CheckAB, Return (1.0, 0, T)
- checkSeedHashes(seedHash_, skA_.getSeedHash());//lgtm [java/dereferenced-value-may-be-null]
- //$FALL-THROUGH$
- case 1: //A Null, B Empty; CheckB, Return (1.0, 0, T)
- case 2: //A Null, B Compact; CheckB, Return (1.0, 0, T)
- case 3: //A Null, B Ordered; CheckB, Return (1.0, 0, T)
- case 4: //A Null, B HashTbl; CheckB, Return (1.0, 0, T)
- checkSeedHashes(seedHash_, skB_.getSeedHash());//lgtm [java/dereferenced-value-may-be-null]
- thetaLong_ = Long.MAX_VALUE;
- empty_ = true;
- break;
-
- case 9: //A Empty, B Empty; CheckAB, Return (1.0, 0, T)
- checkSeedHashes(seedHash_, skB_.getSeedHash());//lgtm [java/dereferenced-value-may-be-null]
- //$FALL-THROUGH$
- case 8: //A Empty, B Null; CheckA, Return (1.0, 0, T)
- checkSeedHashes(seedHash_, skA_.getSeedHash());//lgtm [java/dereferenced-value-may-be-null]
- thetaLong_ = Long.MAX_VALUE;
- empty_ = true;
- break;
-
- case 17: //A Compact, B Empty; CheckAB, Return (ThA, |A|, F), copyA
- case 25: //A Ordered, B Empty; CheckAB, Return (ThA, |A|, F), copyA
- case 33: //A HashTbl, B Empty; CheckAB, Return (ThA, |A|, F), copyA
- checkSeedHashes(seedHash_, skB_.getSeedHash());//lgtm [java/dereferenced-value-may-be-null]
- //$FALL-THROUGH$
- case 16: //A Compact, B Null; CheckA, Return (ThA, |A|, F), copyA
- case 24: //A Ordered, B Null; CheckA, Return (ThA, |A|, F), copyA
- case 32: //A HashTbl, B Null; CheckA, Return (ThA, |A|, F), copyA
- checkSeedHashes(seedHash_, skA_.getSeedHash());//lgtm [java/dereferenced-value-may-be-null]
- thetaLong_ = skA_.getThetaLong();
- empty_ = false;
- curCount_ = skA_.getRetainedEntries(true);
- hashArr_ = CompactOperations.compactCache(skA_.getCache(), curCount_, thetaLong_, false);
- break;
-
- case 18: //A Compact, B Compact; CheckAB, B -> H; => C,H; scanAllAsearchB()
- case 19: //A Compact, B Ordered; CheckAB, B -> H; => C,H; scanAllAsearchB()
- case 34: //A HashTbl, B Compact; CheckAB, B -> H; => H,H; scanAllAsearchB()
- case 35: //A HashTbl, B Ordered; CheckAB, B -> H; => H,H; scanAllAsearchB()
- checkSeedHashes(seedHash_, skA_.getSeedHash());//lgtm [java/dereferenced-value-may-be-null]
- checkSeedHashes(seedHash_, skB_.getSeedHash());//lgtm [java/dereferenced-value-may-be-null]
- thetaLong_ = min(skA_.getThetaLong(), skB_.getThetaLong());
- empty_ = false;
- convertBtoHT();
- scanAllAsearchB();
- break;
-
- case 26: //A Ordered, B Compact; CheckAB, B -> H; => O,H; scanEarlyStopAsearchB()
- case 27: //A Ordered, B Ordered; CheckAB, B -> H; => O,H; scanEarlyStopAsearchB()
- checkSeedHashes(seedHash_, skA_.getSeedHash());//lgtm [java/dereferenced-value-may-be-null]
- checkSeedHashes(seedHash_, skB_.getSeedHash());//lgtm [java/dereferenced-value-may-be-null]
- thetaLong_ = min(skA_.getThetaLong(), skB_.getThetaLong());
- empty_ = false;
- convertBtoHT();
- scanEarlyStopAsearchB();
- break;
-
- case 20: //A Compact, B HashTbl; CheckAB, scanAllAsearchB()
- case 36: //A HashTbl, B HashTbl; CheckAB, scanAllAsearchB()
- checkSeedHashes(seedHash_, skA_.getSeedHash());//lgtm [java/dereferenced-value-may-be-null]
- checkSeedHashes(seedHash_, skB_.getSeedHash());//lgtm [java/dereferenced-value-may-be-null]
- thetaLong_ = min(skA_.getThetaLong(), skB_.getThetaLong());
- empty_ = false;
- lgArrLongsHT_ = ((UpdateSketch)skB_).getLgArrLongs();
- bHashTable_ = skB_.getCache();
- scanAllAsearchB();
- break;
-
- case 28: //A Ordered, B HashTbl; CheckAB, scanEarlyStopAsearchB()
- checkSeedHashes(seedHash_, skA_.getSeedHash());//lgtm [java/dereferenced-value-may-be-null]
- checkSeedHashes(seedHash_, skB_.getSeedHash());//lgtm [java/dereferenced-value-may-be-null]
- thetaLong_ = min(skA_.getThetaLong(), skB_.getThetaLong());
- empty_ = false;
- lgArrLongsHT_ = ((UpdateSketch)skB_).getLgArrLongs();
- bHashTable_ = skB_.getCache();
- scanEarlyStopAsearchB();
- break;
-
- //default: //This cannot happen and cannot be tested
- }
- }
-
- private void convertBtoHT() {
- final int curCountB = skB_.getRetainedEntries(true);
- lgArrLongsHT_ = computeMinLgArrLongsFromCount(curCountB);
- bHashTable_ = new long[1 << lgArrLongsHT_];
- hashArrayInsert(skB_.getCache(), bHashTable_, lgArrLongsHT_, thetaLong_);
- }
-
- //Sketch A is either unordered compact or hash table
- private void scanAllAsearchB() {
- final long[] scanAArr = skA_.getCache();
- final int arrLongsIn = scanAArr.length;
- hashArr_ = new long[arrLongsIn];
- for (int i = 0; i < arrLongsIn; i++ ) {
- final long hashIn = scanAArr[i];
- if ((hashIn <= 0L) || (hashIn >= thetaLong_)) { continue; }
- final int foundIdx = hashSearch(bHashTable_, lgArrLongsHT_, hashIn);
- if (foundIdx > -1) { continue; }
- hashArr_[curCount_++] = hashIn;
- }
- }
-
- //Sketch A is ordered compact, which enables early stop
- private void scanEarlyStopAsearchB() {
- final long[] scanAArr = skA_.getCache();
- final int arrLongsIn = scanAArr.length;
- hashArr_ = new long[arrLongsIn]; //maybe 2x what is needed, but getRetainedEntries can be slow.
- for (int i = 0; i < arrLongsIn; i++ ) {
- final long hashIn = scanAArr[i];
- if (hashIn <= 0L) { continue; }
- if (hashIn >= thetaLong_) {
- break; //early stop assumes that hashes in input sketch are ordered!
- }
- final int foundIdx = hashSearch(bHashTable_, lgArrLongsHT_, hashIn);
- if (foundIdx > -1) { continue; }
- hashArr_[curCount_++] = hashIn;
- }
- }
-
private void reset() {
- skA_ = null;
- skB_ = null;
thetaLong_ = Long.MAX_VALUE;
empty_ = true;
hashArr_ = null;
curCount_ = 0;
- lgArrLongsHT_ = 5;
- bHashTable_ = null;
}
@Override
@@ -524,4 +235,26 @@ final class AnotBimpl extends AnotB {
return thetaLong_;
}
+ //Deprecated methods
+
+ @Deprecated
+ @Override
+ public void update(final Sketch skA, final Sketch skB) {
+ reset();
+ setA(skA);
+ notB(skB);
+ }
+
+ @Deprecated
+ @Override
+ public CompactSketch getResult() {
+ return getResult(true, null);
+ }
+
+ @Deprecated
+ @Override
+ public CompactSketch getResult(final boolean dstOrdered, final WritableMemory dstMem) {
+ return getResult(dstOrdered, dstMem, true);
+ }
+
}
diff --git a/src/main/java/org/apache/datasketches/theta/IntersectionImplR.java b/src/main/java/org/apache/datasketches/theta/IntersectionImplR.java
index 6b79a24..65359d2 100644
--- a/src/main/java/org/apache/datasketches/theta/IntersectionImplR.java
+++ b/src/main/java/org/apache/datasketches/theta/IntersectionImplR.java
@@ -173,7 +173,6 @@ class IntersectionImplR extends Intersection {
}
compactCacheR = compactCachePart(hashTable, lgArrLongs_, curCount_, thetaLong_, dstOrdered);
- //Create the CompactSketch
return CompactOperations.componentsToCompact(
thetaLong_, curCount_, seedHash_, empty_, true, dstOrdered, dstOrdered, dstMem, compactCacheR);
}
diff --git a/src/main/java/org/apache/datasketches/theta/PairwiseSetOperations.java b/src/main/java/org/apache/datasketches/theta/PairwiseSetOperations.java
index 3ffa6ac..fc24f53 100644
--- a/src/main/java/org/apache/datasketches/theta/PairwiseSetOperations.java
+++ b/src/main/java/org/apache/datasketches/theta/PairwiseSetOperations.java
@@ -20,13 +20,6 @@
package org.apache.datasketches.theta;
import static org.apache.datasketches.Util.DEFAULT_NOMINAL_ENTRIES;
-import static org.apache.datasketches.Util.DEFAULT_UPDATE_SEED;
-import static org.apache.datasketches.Util.checkSeedHashes;
-import static org.apache.datasketches.Util.computeSeedHash;
-
-import java.util.Arrays;
-
-import org.apache.datasketches.SketchesArgumentException;
/**
* Set Operations where the arguments are presented in pairs as in <i>C = Op(A,B)</i>. These are
@@ -37,7 +30,7 @@ import org.apache.datasketches.SketchesArgumentException;
*
* @author Lee Rhodes
* @deprecated This class has been deprecated as equivalent functionality has been added to the
- * SetOperation classes: Union, Intersection and AnotB.
+ * SetOperation classes: {@link Union}, {@link Intersection} and {@link AnotB}.
*/
@Deprecated
public class PairwiseSetOperations {
@@ -55,13 +48,8 @@ public class PairwiseSetOperations {
*/
@Deprecated
public static CompactSketch intersect(final Sketch skA, final Sketch skB) {
- if (((skA == null) || (skA instanceof EmptyCompactSketch))
- || ((skB == null) || (skB instanceof EmptyCompactSketch))) {
- return EmptyCompactSketch.getInstance();
- }
- final short seedHash = skA.getSeedHash();
- final Intersection inter = new IntersectionImpl(seedHash);
- return inter.intersect(skA, skB, true, null);
+ final Intersection inter = new SetOperationBuilder().buildIntersection();
+ return inter.intersect(skA, skB);
}
/**
@@ -77,14 +65,8 @@ public class PairwiseSetOperations {
*/
@Deprecated
public static CompactSketch aNotB(final Sketch skA, final Sketch skB) {
- if (((skA == null) || (skA instanceof EmptyCompactSketch))
- && ((skB == null) || (skB instanceof EmptyCompactSketch))) {
- return EmptyCompactSketch.getInstance();
- }
- final short seedHash = ((skA == null) || (skA instanceof EmptyCompactSketch))
- ? skB.getSeedHash() : skA.getSeedHash(); // lgtm [java/dereferenced-value-may-be-null]
- final AnotBimpl anotb = new AnotBimpl(seedHash);
- return anotb.aNotB(skA, skB, true, null);
+ final AnotB anotb = new SetOperationBuilder().buildANotB();
+ return anotb.aNotB(skA, skB);
}
/**
@@ -123,133 +105,9 @@ public class PairwiseSetOperations {
* complete seed handling.
*/
@Deprecated
- @SuppressWarnings("null")
public static CompactSketch union(final CompactSketch skA, final CompactSketch skB, final int k) {
- //Handle all corner cases with null or empty arguments
- //For backward compatibility, we must allow input empties with Theta < 1.0.
- final int swA, swB;
- swA = ((skA == null) || (skA instanceof EmptyCompactSketch)) ? 0 : 2;
- swB = ((skB == null) || (skB instanceof EmptyCompactSketch)) ? 0 : 1;
- final int sw = swA | swB;
- switch (sw) {
- case 0: { //skA == null/ECS; skB == null/ECS; return EmptyCompactSketch.
- return EmptyCompactSketch.getInstance();
- }
- case 1: { //skA == null/ECS; skB == valid; return skB
- checkOrdered(skB);
- return maybeCutback(skB, k);
- }
- case 2: { //skA == valid; skB == null/ECS; return skA
- checkOrdered(skA);
- return maybeCutback(skA, k);
- }
- case 3: { //skA == valid; skB == valid; perform full union
- checkOrdered(skA);
- checkOrdered(skB);
- seedHashesCheck(skA, skB);
- break;
- }
- //default: cannot happen
- }
-
- //Both sketches are valid with matching seedhashes and ordered
- //Full Union operation:
- final long thetaLongA = skA.getThetaLong(); //lgtm [java/dereferenced-value-may-be-null]
- final long thetaLongB = skB.getThetaLong(); //lgtm [java/dereferenced-value-may-be-null]
- long thetaLong = Math.min(thetaLongA, thetaLongB); //Theta rule
- final long[] cacheA = (skA.hasMemory()) ? skA.getCache() : skA.getCache().clone();
- final long[] cacheB = (skB.hasMemory()) ? skB.getCache() : skB.getCache().clone();
- final int aLen = cacheA.length;
- final int bLen = cacheB.length;
-
- final long[] outCache = new long[aLen + bLen];
-
- int indexA = 0;
- int indexB = 0;
- int indexOut = 0;
- long hashA = (aLen == 0) ? thetaLong : cacheA[indexA];
- long hashB = (bLen == 0) ? thetaLong : cacheB[indexB];
-
- while ((indexA < aLen) || (indexB < bLen)) {
- if (hashA == hashB) {
- if (hashA < thetaLong) {
- if (indexOut >= k) {
- thetaLong = hashA;
- break;
- }
- outCache[indexOut++] = hashA;
- hashA = (++indexA < aLen) ? cacheA[indexA] : thetaLong;
- hashB = (++indexB < bLen) ? cacheB[indexB] : thetaLong;
- continue;
- }
- break;
- }
- else if (hashA < hashB) {
- if (hashA < thetaLong) {
- if (indexOut >= k) {
- thetaLong = hashA;
- break;
- }
- outCache[indexOut++] = hashA;
- hashA = (++indexA < aLen) ? cacheA[indexA] : thetaLong;
- continue;
- }
- break;
- }
- else { //hashA > hashB
- if (hashB < thetaLong) {
- if (indexOut >= k) {
- thetaLong = hashB;
- break;
- }
- outCache[indexOut++] = hashB;
- hashB = (++indexB < bLen) ? cacheB[indexB] : thetaLong;
- continue;
- }
- break;
- }
- }
-
- int curCount = indexOut;
- final long[] outArr;
- if (indexOut > k) { //unlikely
- outArr = Arrays.copyOf(outCache, k); //cutback to k, just in case
- curCount = k;
- } else {
- outArr = Arrays.copyOf(outCache, curCount); //copy only valid items
- }
- final short seedHash = computeSeedHash(DEFAULT_UPDATE_SEED);
- final boolean srcEmpty = (curCount == 0) && (thetaLong == Long.MAX_VALUE);
- return CompactOperations.componentsToCompact(
- thetaLong, curCount, seedHash, srcEmpty, true, true, true, null, outArr);
- }
-
- private static CompactSketch maybeCutback(final CompactSketch csk, final int k) {
- final boolean empty = csk.isEmpty();
- int curCount = csk.getRetainedEntries(true);
- long thetaLong = csk.getThetaLong();
- if (curCount > k) { //cutback to k
- final long[] cache = (csk.hasMemory()) ? csk.getCache() : csk.getCache().clone();
- thetaLong = cache[k];
- final long[] arr = Arrays.copyOf(cache, k);
- curCount = k;
- final short seedHash = computeSeedHash(DEFAULT_UPDATE_SEED);
- return CompactOperations.componentsToCompact(
- thetaLong, curCount, seedHash, empty, true, false, true, null, cache);
- }
- return csk;
- }
-
- private static void checkOrdered(final CompactSketch csk) {
- if (!csk.isOrdered()) {
- throw new SketchesArgumentException("Given sketch must be ordered.");
- }
- }
-
- private static short seedHashesCheck(final Sketch skA, final Sketch skB) {
- final short seedHashA = skA.getSeedHash(); //lgtm [java/dereferenced-value-may-be-null]
- final short seedHashB = skB.getSeedHash(); //lgtm [java/dereferenced-value-may-be-null]
- return checkSeedHashes(seedHashA, seedHashB);
+ final Union un = new SetOperationBuilder().setNominalEntries(k).buildUnion();
+ return un.union(skA, skB);
}
}
diff --git a/src/main/java/org/apache/datasketches/theta/SetOperation.java b/src/main/java/org/apache/datasketches/theta/SetOperation.java
index 89eed19..a38b719 100644
--- a/src/main/java/org/apache/datasketches/theta/SetOperation.java
+++ b/src/main/java/org/apache/datasketches/theta/SetOperation.java
@@ -271,17 +271,4 @@ public abstract class SetOperation {
return ret;
}
- /**
- * This corrects a temporary anomalous condition from an intersection of two exact, disjoint sets,
- * or AnotB of two exact, identical sets. Because there is no probability error distribuion,
- * the result is exactly empty. The empty flag is changed to true before returning a result.
- * This is used in the compaction step of SetOperation.
- * @param curCount the given curCount
- * @param thetaLong the given theta.
- * @return the empty state
- */
- private static final boolean correctEmptyOnSetResult(final int curCount, final long thetaLong) {
- return (curCount == 0) && (thetaLong == Long.MAX_VALUE);
- }
-
}
diff --git a/src/test/java/org/apache/datasketches/theta/AnotBimplTest.java b/src/test/java/org/apache/datasketches/theta/AnotBimplTest.java
index 3f4ba4d..4341be0 100644
--- a/src/test/java/org/apache/datasketches/theta/AnotBimplTest.java
+++ b/src/test/java/org/apache/datasketches/theta/AnotBimplTest.java
@@ -51,11 +51,12 @@ public class AnotBimplTest {
}
AnotB aNb = SetOperation.builder().buildANotB();
- assertTrue(aNb.isEmpty());
- assertNull(aNb.getCache());
- assertEquals(aNb.getThetaLong(), Long.MAX_VALUE);
+ assertTrue(aNb.isEmpty()); //only applies to stateful
+ assertNull(aNb.getCache()); //only applies to stateful
+ assertEquals(aNb.getThetaLong(), Long.MAX_VALUE); //only applies to stateful
assertEquals(aNb.getSeedHash(), Util.computeSeedHash(DEFAULT_UPDATE_SEED));
+
aNb.update(usk1, usk2);
assertEquals(aNb.getRetainedEntries(true), 256);
diff --git a/src/test/java/org/apache/datasketches/theta/SetOpsCornerCasesTest.java b/src/test/java/org/apache/datasketches/theta/SetOpsCornerCasesTest.java
index 2626760..d29a84d 100644
--- a/src/test/java/org/apache/datasketches/theta/SetOpsCornerCasesTest.java
+++ b/src/test/java/org/apache/datasketches/theta/SetOpsCornerCasesTest.java
@@ -244,32 +244,12 @@ public class SetOpsCornerCasesTest {
CompactSketch skHeap = generate(EST_HEAP, k);
CompactSketch skHeapUO = generate(EST_MEMORY_UNORDERED, k);
- try {
- PairwiseSetOperations.union(skNull, skHeapUO, k);
- Assert.fail();
- } catch (Exception e) { } //pass
- try {
- PairwiseSetOperations.union(skEmpty, skHeapUO, k);
- Assert.fail();
- } catch (Exception e) { } //pass
-
- try {
- PairwiseSetOperations.union(skHeapUO, skNull, k);
- Assert.fail();
- } catch (Exception e) { } //pass
- try {
- PairwiseSetOperations.union(skHeapUO, skEmpty, k);
- Assert.fail();
- } catch (Exception e) { } //pass
-
- try {
- PairwiseSetOperations.union(skHeapUO, skHeap, k);
- Assert.fail();
- } catch (Exception e) { } //pass
- try {
- PairwiseSetOperations.union(skHeap, skHeapUO, k);
- Assert.fail();
- } catch (Exception e) { } //pass
+ PairwiseSetOperations.union(skNull, skHeapUO, k);
+ PairwiseSetOperations.union(skEmpty, skHeapUO, k);
+ PairwiseSetOperations.union(skHeapUO, skNull, k);
+ PairwiseSetOperations.union(skHeapUO, skEmpty, k);
+ PairwiseSetOperations.union(skHeapUO, skHeap, k);
+ PairwiseSetOperations.union(skHeap, skHeapUO, k);
}
@Test
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[incubator-datasketches-java] 10/10: Add unit tests
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leerho pushed a commit to branch Refactor_Theta_Tuple
in repository https://gitbox.apache.org/repos/asf/incubator-datasketches-java.git
commit 644ab859645efce4fb14665533117edbed17bf16
Author: Lee Rhodes <le...@users.noreply.github.com>
AuthorDate: Tue Jul 7 16:35:03 2020 -0700
Add unit tests
---
.../apache/datasketches/theta/IntersectionImpl.java | 6 +-----
.../apache/datasketches/theta/IntersectionImplR.java | 18 +++++++++---------
.../org/apache/datasketches/theta/SetOperation.java | 5 +++--
.../org/apache/datasketches/theta/AnotBimplTest.java | 6 ++++++
.../apache/datasketches/theta/CompactSketchTest.java | 11 +++++++++++
.../java/org/apache/datasketches/theta/EmptyTest.java | 15 +++++++++++++--
.../datasketches/theta/HeapIntersectionTest.java | 2 +-
.../datasketches/theta/SingleItemSketchTest.java | 15 +++++++++++++++
.../java/org/apache/datasketches/theta/SketchTest.java | 11 +++++++++++
9 files changed, 70 insertions(+), 19 deletions(-)
diff --git a/src/main/java/org/apache/datasketches/theta/IntersectionImpl.java b/src/main/java/org/apache/datasketches/theta/IntersectionImpl.java
index 611130d..7dd1368 100644
--- a/src/main/java/org/apache/datasketches/theta/IntersectionImpl.java
+++ b/src/main/java/org/apache/datasketches/theta/IntersectionImpl.java
@@ -55,10 +55,6 @@ final class IntersectionImpl extends IntersectionImplR {
super(wmem, seed, newMem);
}
- IntersectionImpl(final short seedHash) {
- super(seedHash);
- }
-
/**
* Construct a new Intersection target on the java heap.
*
@@ -260,7 +256,7 @@ final class IntersectionImpl extends IntersectionImplR {
performIntersect(sketchIn);
break;
}
- default: assert false : "Should not happen";
+ //default: assert false : "Should not happen";
}
}
diff --git a/src/main/java/org/apache/datasketches/theta/IntersectionImplR.java b/src/main/java/org/apache/datasketches/theta/IntersectionImplR.java
index 65359d2..e63d708 100644
--- a/src/main/java/org/apache/datasketches/theta/IntersectionImplR.java
+++ b/src/main/java/org/apache/datasketches/theta/IntersectionImplR.java
@@ -83,15 +83,15 @@ class IntersectionImplR extends Intersection {
}
}
- IntersectionImplR(final short seedHash) {
- seedHash_ = seedHash;
- mem_ = null;
- lgArrLongs_ = 0;
- curCount_ = -1;
- thetaLong_ = Long.MAX_VALUE;
- empty_ = false;
- hashTable_ = null;
- }
+ // IntersectionImplR(final short seedHash) {
+ // seedHash_ = seedHash;
+ // mem_ = null;
+ // lgArrLongs_ = 0;
+ // curCount_ = -1;
+ // thetaLong_ = Long.MAX_VALUE;
+ // empty_ = false;
+ // hashTable_ = null;
+ // }
/**
* Wrap an Intersection target around the given source Memory containing intersection data.
diff --git a/src/main/java/org/apache/datasketches/theta/SetOperation.java b/src/main/java/org/apache/datasketches/theta/SetOperation.java
index a38b719..b43977c 100644
--- a/src/main/java/org/apache/datasketches/theta/SetOperation.java
+++ b/src/main/java/org/apache/datasketches/theta/SetOperation.java
@@ -204,11 +204,12 @@ public abstract class SetOperation {
/**
* Returns the maximum number of bytes for the returned CompactSketch, given the maximum
* value of nomEntries of the first sketch A of AnotB.
- * @param maxNomEntries the given value
+ * @param maxNomEntries the given value must be a power of 2.
* @return the maximum number of bytes.
*/
public static int getMaxAnotBResultBytes(final int maxNomEntries) {
- return 24 + (15 * maxNomEntries);
+ final int ceil = ceilingPowerOf2(maxNomEntries);
+ return 24 + (15 * ceil);
}
diff --git a/src/test/java/org/apache/datasketches/theta/AnotBimplTest.java b/src/test/java/org/apache/datasketches/theta/AnotBimplTest.java
index 3ad0559..6d20235 100644
--- a/src/test/java/org/apache/datasketches/theta/AnotBimplTest.java
+++ b/src/test/java/org/apache/datasketches/theta/AnotBimplTest.java
@@ -378,6 +378,12 @@ public class AnotBimplTest {
}
@Test
+ public void checkGetMaxBytes() {
+ int bytes = Sketches.getMaxAnotBResultBytes(10);
+ assertEquals(bytes, (16 * 15) + 24);
+ }
+
+ @Test
public void printlnTest() {
println("PRINTING: "+this.getClass().getName());
}
diff --git a/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java b/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
index fe8347e..f8f1811 100644
--- a/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
@@ -323,6 +323,17 @@ public class CompactSketchTest {
assertTrue(cache.length == 0);
}
+ @Test
+ public void checkHeapCompactSketchCompact() {
+ UpdateSketch sk = Sketches.updateSketchBuilder().build();
+ sk.update(1);
+ sk.update(2);
+ CompactSketch csk = sk.compact();
+ CompactSketch csk2 = csk.compact();
+ assertTrue(csk.isOrdered());
+ assertEquals(csk.getCurrentPreambleLongs(), 2);
+ }
+
private static class State {
String classType = null;
int count = 0;
diff --git a/src/test/java/org/apache/datasketches/theta/EmptyTest.java b/src/test/java/org/apache/datasketches/theta/EmptyTest.java
index 7b93117..480421e 100644
--- a/src/test/java/org/apache/datasketches/theta/EmptyTest.java
+++ b/src/test/java/org/apache/datasketches/theta/EmptyTest.java
@@ -136,6 +136,19 @@ public class EmptyTest {
Sketches.heapifySketch(mem);
}
+ @Test
+ public void checkEmptyToCompact() {
+ UpdateSketch sk1 = Sketches.updateSketchBuilder().build();
+ CompactSketch csk = sk1.compact();
+ assertTrue(csk instanceof EmptyCompactSketch);
+ CompactSketch csk2 = csk.compact();
+ assertTrue(csk2 instanceof EmptyCompactSketch);
+ CompactSketch csk3 = csk.compact(true, WritableMemory.allocate(8));
+ assertTrue(csk3 instanceof DirectCompactSketch);
+ assertEquals(csk2.getCurrentPreambleLongs(), 1);
+ }
+
+
//SerVer 2 had an empty sketch where preLongs = 1, but empty bit was not set.
private static Memory badEmptySk() {
final long preLongs = 1;
@@ -150,8 +163,6 @@ public class EmptyTest {
return wmem;
}
-
-
/**
* @param s value to print
*/
diff --git a/src/test/java/org/apache/datasketches/theta/HeapIntersectionTest.java b/src/test/java/org/apache/datasketches/theta/HeapIntersectionTest.java
index 4008203..f4abbca 100644
--- a/src/test/java/org/apache/datasketches/theta/HeapIntersectionTest.java
+++ b/src/test/java/org/apache/datasketches/theta/HeapIntersectionTest.java
@@ -509,7 +509,7 @@ public class HeapIntersectionTest {
byte[] byteArray = union.toByteArray();
Memory mem = Memory.wrap(byteArray);
Intersection inter1 = (Intersection) SetOperation.heapify(mem); //bad cast
- println(inter1.toString());
+ //println(inter1.toString());
}
@Test(expectedExceptions = SketchesArgumentException.class)
diff --git a/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java b/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java
index 01226bf..dbb8088 100644
--- a/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java
@@ -323,6 +323,21 @@ public class SingleItemSketchTest {
//println(sk.toString());
}
+ @Test
+ public void checkSingleItemCompact() {
+ UpdateSketch sk1 = new UpdateSketchBuilder().build();
+ sk1.update(1);
+ CompactSketch csk = sk1.compact();
+ assertTrue(csk instanceof SingleItemSketch);
+ CompactSketch csk2 = csk.compact();
+ assertEquals(csk, csk2);
+ CompactSketch csk3 = csk.compact(true, WritableMemory.allocate(16));
+ assertTrue(csk3 instanceof DirectCompactSketch);
+ assertEquals(csk2.getCurrentPreambleLongs(), 1);
+ assertEquals(csk3.getCurrentPreambleLongs(), 1);
+ }
+
+
static final long SiSkPre0WithSiFlag = 0x93cc3a0000030301L;
static final long SiSkPre0WoutSiFlag = 0x93cc1a0000030301L;
static final long Hash = 0x05a186bdcb7df915L;
diff --git a/src/test/java/org/apache/datasketches/theta/SketchTest.java b/src/test/java/org/apache/datasketches/theta/SketchTest.java
index 1193e38..18c7ed7 100644
--- a/src/test/java/org/apache/datasketches/theta/SketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/SketchTest.java
@@ -409,7 +409,18 @@ public class SketchTest {
sk = Sketch.heapify(wmem);
fail();
} catch (SketchesArgumentException e) { }
+ }
+ @Test
+ public void check2Methods() {
+ int k = 16;
+ Sketch sk = Sketches.updateSketchBuilder().setNominalEntries(k).build();
+ int bytes1 = sk.getCurrentBytes(true);
+ int bytes2 = sk.getCurrentBytes(false);
+ assertEquals(bytes1, 8);
+ assertEquals(bytes2, 280); //32*8 + 24
+ int retEnt = sk.getRetainedEntries();
+ assertEquals(retEnt, 0);
}
@Test
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[incubator-datasketches-java] 03/10: Consolidated all compacting
operations into CompactOperations.
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leerho pushed a commit to branch Refactor_Theta_Tuple
in repository https://gitbox.apache.org/repos/asf/incubator-datasketches-java.git
commit f4317e5b0d0925e0cc1462cbac0648cf4b97ae9e
Author: Lee Rhodes <le...@users.noreply.github.com>
AuthorDate: Sun Jun 28 09:20:36 2020 -0700
Consolidated all compacting operations into CompactOperations.
---
.../datasketches/theta/CompactOperations.java | 95 +++++++++++++++++-----
.../datasketches/theta/DirectCompactSketch.java | 6 +-
.../theta/DirectQuickSelectSketchR.java | 4 +-
.../datasketches/theta/ForwardCompatibility.java | 6 +-
.../theta/HeapCompactOrderedSketch.java | 70 ----------------
.../datasketches/theta/HeapCompactSketch.java | 63 ++++++++++----
.../theta/HeapCompactUnorderedSketch.java | 70 ----------------
.../datasketches/theta/HeapUpdateSketch.java | 5 +-
.../java/org/apache/datasketches/theta/Sketch.java | 54 ------------
.../apache/datasketches/theta/UpdateSketch.java | 88 --------------------
.../datasketches/theta/CompactSketchTest.java | 6 +-
.../ConcurrentDirectQuickSelectSketchTest.java | 4 +-
.../theta/ConcurrentHeapQuickSelectSketchTest.java | 4 +-
.../theta/DirectQuickSelectSketchTest.java | 4 +-
.../theta/ForwardCompatibilityTest.java | 2 +-
.../datasketches/theta/HeapAlphaSketchTest.java | 4 +-
.../theta/HeapQuickSelectSketchTest.java | 4 +-
17 files changed, 146 insertions(+), 343 deletions(-)
diff --git a/src/main/java/org/apache/datasketches/theta/CompactOperations.java b/src/main/java/org/apache/datasketches/theta/CompactOperations.java
index 882e0e8..2c17ade 100644
--- a/src/main/java/org/apache/datasketches/theta/CompactOperations.java
+++ b/src/main/java/org/apache/datasketches/theta/CompactOperations.java
@@ -100,11 +100,7 @@ final class CompactOperations {
if (single) {
return new SingleItemSketch(hashArrOut[0], seedHash);
}
- if (dstOrderedOut) {
- return new HeapCompactOrderedSketch(hashArrOut, empty, seedHash, curCount, thetaLong);
- } else {
- return new HeapCompactUnorderedSketch(hashArrOut, empty, seedHash, curCount, thetaLong);
- }
+ return new HeapCompactSketch(hashArrOut, empty, seedHash, curCount, thetaLong, dstOrderedOut);
}
}
@@ -116,7 +112,7 @@ final class CompactOperations {
* @param dstMem Used for the target CompactSketch if it is Direct.
* @return a CompactSketch of the correct form.
*/
- @SuppressWarnings("unused") //to replace CompactSketch.anyMemoryToCompactHeap
+ @SuppressWarnings("unused")
static CompactSketch memoryToCompact(
final Memory srcMem,
final boolean dstOrdered,
@@ -136,10 +132,10 @@ final class CompactOperations {
final boolean srcEmptyFlag = (srcFlags & EMPTY_FLAG_MASK) > 0;
final boolean srcCompactFlag = (srcFlags & COMPACT_FLAG_MASK) > 0;
final boolean srcOrderedFlag = (srcFlags & ORDERED_FLAG_MASK) > 0;
- //final boolean srcSingleFlag = (srcFlags & SINGLEITEM_FLAG_MASK) > 0;
+ final boolean srcSingleFlag = (srcFlags & SINGLEITEM_FLAG_MASK) > 0;
- final boolean single =
- SingleItemSketch.otherCheckForSingleItem(srcPreLongs, srcSerVer, srcFamId, srcFlags);
+ final boolean single = srcSingleFlag
+ || SingleItemSketch.otherCheckForSingleItem(srcPreLongs, srcSerVer, srcFamId, srcFlags);
//extract pre1 and pre2 fields
final int curCount = single ? 1 : (srcPreLongs > 1) ? extractCurCount(srcMem) : 0;
@@ -189,13 +185,8 @@ final class CompactOperations {
final Memory tgtMem = loadCompactMemory(hashArr, srcSeedHash, curCount, thetaLong, dstMem,
(byte)srcFlags, srcPreLongs);
return new DirectCompactSketch(tgtMem);
-
} else { //heap
- if (dstOrderedOut) {
- return new HeapCompactOrderedSketch(hashArr, srcEmptyFlag, srcSeedHash, curCount, thetaLong);
- } else {
- return new HeapCompactUnorderedSketch(hashArr, srcEmptyFlag, srcSeedHash, curCount, thetaLong);
- }
+ return new HeapCompactSketch(hashArr, srcEmptyFlag, srcSeedHash, curCount, thetaLong, dstOrderedOut);
}
}
@@ -262,11 +253,6 @@ final class CompactOperations {
return dstMem; //curCount == 0, theta could be < 1.0
}
- static final int computeCompactPreLongs(final long thetaLong, final boolean empty,
- final int curCount) {
- return (thetaLong < Long.MAX_VALUE) ? 3 : empty ? 1 : (curCount > 1) ? 2 : 1;
- }
-
/**
* Copies then compacts, cleans, and may sort the resulting array.
* The source cache can be a hash table with interstitial zeros or
@@ -302,5 +288,74 @@ final class CompactOperations {
return cacheOut;
}
+ /*
+ * The truth table for empty, curCount and theta when compacting is as follows:
+ * <pre>
+ * Num Theta CurCount Empty State Comments
+ * 0 1.0 0 T OK The Normal Empty State
+ * 1 1.0 0 F Internal This can result from an intersection of two exact, disjoint sets,
+ * or AnotB of two exact, identical sets. There is no probability
+ * distribution, so change to empty. Return {Th = 1.0, 0, T}.
+ * This is handled in SetOperation.createCompactSketch().
+ * 2 1.0 !0 T Error Empty=T and curCount !0 should never co-exist.
+ * This is checked in all compacting operations.
+ * 3 1.0 !0 F OK This corresponds to a sketch in exact mode
+ * 4 <1.0 0 T Internal This can be an initial UpdateSketch state if p < 1.0,
+ * so change theta to 1.0. Return {Th = 1.0, 0, T}.
+ * This is handled in UpdateSketch.compact() and toByteArray().
+ * 5 <1.0 0 F OK This can result from set operations
+ * 6 <1.0 !0 T Error Empty=T and curCount !0 should never co-exist.
+ * This is checked in all compacting operations.
+ * 7 <1.0 !0 F OK This corresponds to a sketch in estimation mode
+ * </pre>
+ * #4 is handled by <i>correctThetaOnCompat(boolean, int)</i> (below).
+ * #2 & #6 handled by <i>checkIllegalCurCountAndEmpty(boolean, int)</i>
+ */
+
+ /**
+ * This corrects a temporary anomalous condition where compact() is called on an UpdateSketch
+ * that was initialized with p < 1.0 and update() was never called. In this case Theta < 1.0,
+ * curCount = 0, and empty = true. The correction is to change Theta to 1.0, which makes the
+ * returning sketch empty. This should only be used in the compaction or serialization of an
+ * UpdateSketch.
+ * @param empty the given empty state
+ * @param curCount the given curCount
+ * @param thetaLong the given thetaLong
+ * @return thetaLong
+ */
+ static final long correctThetaOnCompact(final boolean empty, final int curCount,
+ final long thetaLong) { //handles #4 above
+ return (empty && (curCount == 0)) ? Long.MAX_VALUE : thetaLong;
+ }
+
+ /**
+ * This checks for the illegal condition where curCount > 0 and the state of
+ * empty = true. This check can be used anywhere a sketch is returned or a sketch is created
+ * from complete arguments.
+ * @param empty the given empty state
+ * @param curCount the given current count
+ */ //This handles #2 and #6 above
+ static final void checkIllegalCurCountAndEmpty(final boolean empty, final int curCount) {
+ if (empty && (curCount != 0)) { //this handles #2 and #6 above
+ throw new SketchesStateException("Illegal State: Empty=true and Current Count != 0.");
+ }
+ }
+
+ static final int computeCompactPreLongs(final long thetaLong, final boolean empty,
+ final int curCount) {
+ return (thetaLong < Long.MAX_VALUE) ? 3 : empty ? 1 : (curCount > 1) ? 2 : 1;
+ }
+
+ /**
+ * This checks for the singleItem Compact Sketch.
+ * @param empty the given empty state
+ * @param curCount the given curCount
+ * @param thetaLong the given thetaLong
+ * @return true if notEmpty, curCount = 1 and theta = 1.0;
+ */
+ static final boolean isSingleItem(final boolean empty, final int curCount,
+ final long thetaLong) {
+ return !empty && (curCount == 1) && (thetaLong == Long.MAX_VALUE);
+ }
}
diff --git a/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java b/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java
index a695944..aac4431 100644
--- a/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java
@@ -19,6 +19,8 @@
package org.apache.datasketches.theta;
+import static org.apache.datasketches.theta.CompactOperations.checkIllegalCurCountAndEmpty;
+import static org.apache.datasketches.theta.CompactOperations.memoryToCompact;
import static org.apache.datasketches.theta.PreambleUtil.ORDERED_FLAG_MASK;
import static org.apache.datasketches.theta.PreambleUtil.checkMemorySeedHash;
import static org.apache.datasketches.theta.PreambleUtil.extractCurCount;
@@ -76,7 +78,7 @@ class DirectCompactSketch extends CompactSketch {
@Override
public CompactSketch compact(final boolean dstOrdered, final WritableMemory dstMem) {
- return CompactOperations.memoryToCompact(mem_, dstOrdered, dstMem);
+ return memoryToCompact(mem_, dstOrdered, dstMem);
}
@Override
@@ -146,7 +148,7 @@ class DirectCompactSketch extends CompactSketch {
@Override
public byte[] toByteArray() {
final int curCount = getRetainedEntries(true);
- Sketch.checkIllegalCurCountAndEmpty(isEmpty(), curCount);
+ checkIllegalCurCountAndEmpty(isEmpty(), curCount);
final int preLongs = extractPreLongs(mem_);
final int outBytes = (curCount + preLongs) << 3;
final byte[] byteArrOut = new byte[outBytes];
diff --git a/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java b/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java
index 97eb82a..f5bfb49 100644
--- a/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java
+++ b/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java
@@ -170,7 +170,7 @@ class DirectQuickSelectSketchR extends UpdateSketch {
@Override
public byte[] toByteArray() { //MY_FAMILY is stored in wmem_
- Sketch.checkIllegalCurCountAndEmpty(isEmpty(), extractCurCount(wmem_));
+ CompactOperations.checkIllegalCurCountAndEmpty(isEmpty(), extractCurCount(wmem_));
final byte lgArrLongs = wmem_.getByte(LG_ARR_LONGS_BYTE);
final int preambleLongs = wmem_.getByte(PREAMBLE_LONGS_BYTE) & 0X3F;
final int lengthBytes = (preambleLongs + (1 << lgArrLongs)) << 3;
@@ -178,7 +178,7 @@ class DirectQuickSelectSketchR extends UpdateSketch {
final WritableMemory mem = WritableMemory.wrap(byteArray);
wmem_.copyTo(0, mem, 0, lengthBytes);
final long thetaLong =
- Sketch.correctThetaOnCompact(isEmpty(), extractCurCount(wmem_), extractThetaLong(wmem_));
+ CompactOperations.correctThetaOnCompact(isEmpty(), extractCurCount(wmem_), extractThetaLong(wmem_));
PreambleUtil.insertThetaLong(wmem_, thetaLong);
return byteArray;
}
diff --git a/src/main/java/org/apache/datasketches/theta/ForwardCompatibility.java b/src/main/java/org/apache/datasketches/theta/ForwardCompatibility.java
index 733dfc1..9da0b56 100644
--- a/src/main/java/org/apache/datasketches/theta/ForwardCompatibility.java
+++ b/src/main/java/org/apache/datasketches/theta/ForwardCompatibility.java
@@ -100,7 +100,7 @@ final class ForwardCompatibility {
final long[] compactOrderedCache = new long[curCount];
srcMem.getLongArray(preLongs << 3, compactOrderedCache, 0, curCount);
- return new HeapCompactOrderedSketch(compactOrderedCache, false, seedHash, curCount, thetaLong);
+ return new HeapCompactSketch(compactOrderedCache, false, seedHash, curCount, thetaLong, true);
}
/**
@@ -146,7 +146,7 @@ final class ForwardCompatibility {
validateInputSize(reqBytesIn, memCap);
final long[] compactOrderedCache = new long[curCount];
srcMem.getLongArray(preLongs << 3, compactOrderedCache, 0, curCount);
- return new HeapCompactOrderedSketch(compactOrderedCache, false, seedHash, curCount, thetaLong);
+ return new HeapCompactSketch(compactOrderedCache, false, seedHash, curCount, thetaLong,true);
}
if (preLongs == 3) { //pre0 + count + theta
reqBytesIn = (preLongs) << 3; //
@@ -167,7 +167,7 @@ final class ForwardCompatibility {
validateInputSize(reqBytesIn, memCap);
final long[] compactOrderedCache = new long[curCount];
srcMem.getLongArray(preLongs << 3, compactOrderedCache, 0, curCount);
- return new HeapCompactOrderedSketch(compactOrderedCache, false, seedHash, curCount, thetaLong);
+ return new HeapCompactSketch(compactOrderedCache, false, seedHash, curCount, thetaLong, true);
}
throw new SketchesArgumentException("PreLongs must be 1,2, or 3: " + preLongs);
}
diff --git a/src/main/java/org/apache/datasketches/theta/HeapCompactOrderedSketch.java b/src/main/java/org/apache/datasketches/theta/HeapCompactOrderedSketch.java
deleted file mode 100644
index dbaf823..0000000
--- a/src/main/java/org/apache/datasketches/theta/HeapCompactOrderedSketch.java
+++ /dev/null
@@ -1,70 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.theta;
-
-import org.apache.datasketches.memory.WritableMemory;
-
-/**
- * An on-heap, compact, ordered, read-only sketch.
- *
- * @author Lee Rhodes
- */
-final class HeapCompactOrderedSketch extends HeapCompactSketch {
-
- /**
- * Constructs this sketch from correct, valid components.
- * @param cache in compact form
- * @param empty The correct <a href="{@docRoot}/resources/dictionary.html#empty">Empty</a>.
- * @param seedHash The correct
- * <a href="{@docRoot}/resources/dictionary.html#seedHash">Seed Hash</a>.
- * @param curCount correct value
- * @param thetaLong The correct
- * <a href="{@docRoot}/resources/dictionary.html#thetaLong">thetaLong</a>.
- */
- HeapCompactOrderedSketch(final long[] cache, final boolean empty, final short seedHash,
- final int curCount, final long thetaLong) {
- super(cache, empty, seedHash, curCount, thetaLong);
- }
-
- //Sketch interface
-
- @Override
- public CompactSketch compact() {
- return this;
- }
-
- @Override
- public CompactSketch compact(final boolean dstOrdered, final WritableMemory dstMem) {
- return CompactOperations.componentsToCompact(
- getThetaLong(), getRetainedEntries(), getSeedHash(), isEmpty(),
- true, true, dstOrdered, dstMem, getCache());
- }
-
- @Override
- public byte[] toByteArray() {
- return toByteArray(true); //ordered
- }
-
- @Override
- public boolean isOrdered() {
- return true;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/theta/HeapCompactSketch.java b/src/main/java/org/apache/datasketches/theta/HeapCompactSketch.java
index 74dff7d..00f8939 100644
--- a/src/main/java/org/apache/datasketches/theta/HeapCompactSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/HeapCompactSketch.java
@@ -19,10 +19,17 @@
package org.apache.datasketches.theta;
+import static org.apache.datasketches.theta.CompactOperations.checkIllegalCurCountAndEmpty;
+import static org.apache.datasketches.theta.CompactOperations.componentsToCompact;
+import static org.apache.datasketches.theta.CompactOperations.computeCompactPreLongs;
+import static org.apache.datasketches.theta.CompactOperations.correctThetaOnCompact;
+import static org.apache.datasketches.theta.CompactOperations.isSingleItem;
+import static org.apache.datasketches.theta.CompactOperations.loadCompactMemory;
import static org.apache.datasketches.theta.PreambleUtil.COMPACT_FLAG_MASK;
import static org.apache.datasketches.theta.PreambleUtil.EMPTY_FLAG_MASK;
import static org.apache.datasketches.theta.PreambleUtil.ORDERED_FLAG_MASK;
import static org.apache.datasketches.theta.PreambleUtil.READ_ONLY_FLAG_MASK;
+import static org.apache.datasketches.theta.PreambleUtil.SINGLEITEM_FLAG_MASK;
import org.apache.datasketches.memory.Memory;
import org.apache.datasketches.memory.WritableMemory;
@@ -32,13 +39,15 @@ import org.apache.datasketches.memory.WritableMemory;
*
* @author Lee Rhodes
*/
-abstract class HeapCompactSketch extends CompactSketch {
+class HeapCompactSketch extends CompactSketch {
+ private final long thetaLong_; //computed
+ private final int curCount_;
+ private final int preLongs_; //computed
private final short seedHash_;
private final boolean empty_;
- private final int curCount_;
- private final long thetaLong_;
+ private final boolean ordered_;
+ private final boolean singleItem_;
private final long[] cache_;
- private final int preLongs_;
/**
* Constructs this sketch from correct, valid components.
@@ -51,18 +60,33 @@ abstract class HeapCompactSketch extends CompactSketch {
* <a href="{@docRoot}/resources/dictionary.html#thetaLong">thetaLong</a>.
*/
HeapCompactSketch(final long[] cache, final boolean empty, final short seedHash,
- final int curCount, final long thetaLong) {
- empty_ = empty;
+ final int curCount, final long thetaLong, final boolean ordered) {
seedHash_ = seedHash;
- curCount_ = empty ? 0 : curCount;
- thetaLong_ = empty ? Long.MAX_VALUE : thetaLong;
+ curCount_ = curCount;
+ empty_ = empty;
+ ordered_ = ordered;
cache_ = cache;
- preLongs_ = CompactOperations.computeCompactPreLongs(thetaLong, empty, curCount);
+ //computed
+ thetaLong_ = correctThetaOnCompact(empty, curCount, thetaLong);
+ preLongs_ = computeCompactPreLongs(thetaLong, empty, curCount); //considers singleItem
+ singleItem_ = isSingleItem(empty, curCount, thetaLong);
+ checkIllegalCurCountAndEmpty(empty, curCount);
}
//Sketch
@Override
+ public CompactSketch compact() {
+ return compact(true, null);
+ }
+
+ @Override
+ public CompactSketch compact(final boolean dstOrdered, final WritableMemory dstMem) {
+ return componentsToCompact(getThetaLong(), getRetainedEntries(), getSeedHash(), isEmpty(),
+ true, ordered_, dstOrdered, dstMem, getCache());
+ }
+
+ @Override
public int getCurrentBytes(final boolean compact) { //already compact; ignored
return (preLongs_ + curCount_) << 3;
}
@@ -98,6 +122,11 @@ abstract class HeapCompactSketch extends CompactSketch {
}
@Override
+ public boolean isOrdered() {
+ return ordered_;
+ }
+
+ @Override
public HashIterator iterator() {
return new HeapHashIterator(cache_, cache_.length, thetaLong_);
}
@@ -124,17 +153,19 @@ abstract class HeapCompactSketch extends CompactSketch {
return seedHash_;
}
- //only called from sub classes
- byte[] toByteArray(final boolean ordered) {
- Sketch.checkIllegalCurCountAndEmpty(empty_, curCount_);
+ //use of Memory is convenient. The byteArray and Memory are loaded simulaneously.
+ @Override
+ public byte[] toByteArray() {
final int bytes = getCurrentBytes(true);
final byte[] byteArray = new byte[bytes];
final WritableMemory dstMem = WritableMemory.wrap(byteArray);
- final int emptyBit = isEmpty() ? (byte) EMPTY_FLAG_MASK : 0;
- final int orderedBit = ordered ? (byte) ORDERED_FLAG_MASK : 0;
- final byte flags = (byte) (emptyBit | READ_ONLY_FLAG_MASK | COMPACT_FLAG_MASK | orderedBit);
+ final int emptyBit = isEmpty() ? EMPTY_FLAG_MASK : 0;
+ final int orderedBit = ordered_ ? ORDERED_FLAG_MASK : 0;
+ final int singleItemBit = singleItem_ ? SINGLEITEM_FLAG_MASK : 0;
+ final byte flags = (byte) (emptyBit | READ_ONLY_FLAG_MASK | COMPACT_FLAG_MASK
+ | orderedBit | singleItemBit);
final int preLongs = getCurrentPreambleLongs(true);
- CompactOperations.loadCompactMemory(getCache(), getSeedHash(), getRetainedEntries(true), getThetaLong(),
+ loadCompactMemory(getCache(), getSeedHash(), getRetainedEntries(true), getThetaLong(),
dstMem, flags, preLongs);
return byteArray;
}
diff --git a/src/main/java/org/apache/datasketches/theta/HeapCompactUnorderedSketch.java b/src/main/java/org/apache/datasketches/theta/HeapCompactUnorderedSketch.java
deleted file mode 100644
index 6b59a02..0000000
--- a/src/main/java/org/apache/datasketches/theta/HeapCompactUnorderedSketch.java
+++ /dev/null
@@ -1,70 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.theta;
-
-import org.apache.datasketches.memory.WritableMemory;
-
-/**
- * An on-heap, compact, unordered, read-only sketch.
- *
- * @author Lee Rhodes
- */
-final class HeapCompactUnorderedSketch extends HeapCompactSketch {
-
- /**
- * Constructs this sketch from correct, valid components.
- * @param cache in compact form
- * @param empty The correct <a href="{@docRoot}/resources/dictionary.html#empty">Empty</a>.
- * @param seedHash The correct
- * <a href="{@docRoot}/resources/dictionary.html#seedHash">Seed Hash</a>.
- * @param curCount correct value
- * @param thetaLong The correct
- * <a href="{@docRoot}/resources/dictionary.html#thetaLong">thetaLong</a>.
- */
- HeapCompactUnorderedSketch(final long[] cache, final boolean empty, final short seedHash,
- final int curCount, final long thetaLong) {
- super(cache, empty, seedHash, curCount, thetaLong);
- }
-
- //Sketch interface
-
- @Override //ordered, on-heap
- public CompactSketch compact() {
- return compact(true, null);
- }
-
- @Override
- public CompactSketch compact(final boolean dstOrdered, final WritableMemory dstMem) {
- return CompactOperations.componentsToCompact(
- getThetaLong(), getRetainedEntries(), getSeedHash(), isEmpty(),
- true, false, dstOrdered, dstMem, getCache());
- }
-
- @Override
- public byte[] toByteArray() {
- return toByteArray(false); //unordered
- }
-
- @Override
- public boolean isOrdered() {
- return false;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java b/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java
index 16efd44..5c642ec 100644
--- a/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java
@@ -106,7 +106,7 @@ abstract class HeapUpdateSketch extends UpdateSketch {
byte[] toByteArray(final int preLongs, final byte familyID) {
if (isDirty()) { rebuild(); }
- Sketch.checkIllegalCurCountAndEmpty(isEmpty(), getRetainedEntries(true));
+ CompactOperations.checkIllegalCurCountAndEmpty(isEmpty(), getRetainedEntries(true));
final int preBytes = (preLongs << 3) & 0X3F;
final int dataBytes = getCurrentDataLongs(false) << 3;
final byte[] byteArrOut = new byte[preBytes + dataBytes];
@@ -124,7 +124,8 @@ abstract class HeapUpdateSketch extends UpdateSketch {
insertCurCount(memOut, this.getRetainedEntries(true));
insertP(memOut, getP());
- final long thetaLong = correctThetaOnCompact(isEmpty(), getRetainedEntries(), getThetaLong());
+ final long thetaLong =
+ CompactOperations.correctThetaOnCompact(isEmpty(), getRetainedEntries(), getThetaLong());
insertThetaLong(memOut, thetaLong);
//Flags: BigEnd=0, ReadOnly=0, Empty=X, compact=0, ordered=0
diff --git a/src/main/java/org/apache/datasketches/theta/Sketch.java b/src/main/java/org/apache/datasketches/theta/Sketch.java
index b8c7bd7..e87b515 100644
--- a/src/main/java/org/apache/datasketches/theta/Sketch.java
+++ b/src/main/java/org/apache/datasketches/theta/Sketch.java
@@ -39,7 +39,6 @@ import static org.apache.datasketches.theta.SingleItemSketch.otherCheckForSingle
import org.apache.datasketches.BinomialBoundsN;
import org.apache.datasketches.Family;
import org.apache.datasketches.SketchesArgumentException;
-import org.apache.datasketches.SketchesStateException;
import org.apache.datasketches.memory.Memory;
import org.apache.datasketches.memory.WritableMemory;
@@ -610,59 +609,6 @@ public abstract class Sketch {
}
}
- /*
- * The truth table for empty, curCount and theta when compacting is as follows:
- * <pre>
- * Num Theta CurCount Empty State Comments
- * 0 1.0 0 T OK The Normal Empty State
- * 1 1.0 0 F Internal This can result from an intersection of two exact, disjoint sets,
- * or AnotB of two exact, identical sets. There is no probability
- * distribution, so change to empty. Return {Th = 1.0, 0, T}.
- * This is handled in SetOperation.createCompactSketch().
- * 2 1.0 !0 T Error Empty=T and curCount !0 should never co-exist.
- * This is checked in all compacting operations.
- * 3 1.0 !0 F OK This corresponds to a sketch in exact mode
- * 4 <1.0 0 T Internal This can be an initial UpdateSketch state if p < 1.0,
- * so change theta to 1.0. Return {Th = 1.0, 0, T}.
- * This is handled in UpdateSketch.compact() and toByteArray().
- * 5 <1.0 0 F OK This can result from set operations
- * 6 <1.0 !0 T Error Empty=T and curCount !0 should never co-exist.
- * This is checked in all compacting operations.
- * 7 <1.0 !0 F OK This corresponds to a sketch in estimation mode
- * </pre>
- */
-
- /**
- * This checks for the illegal condition where curCount > 0 and the state of
- * empty = true. This check can be used anywhere a sketch is returned or a sketch is created
- * from complete arguments.
- * @param empty the given empty state
- * @param curCount the given current count
- */ //This handles #2 and #6 above
- static final void checkIllegalCurCountAndEmpty(final boolean empty, final int curCount) {
- if (empty && (curCount != 0)) { //this handles #2 and #6 above
- throw new SketchesStateException("Illegal State: Empty=true and Current Count != 0.");
- }
- }
-
- /**
- * This corrects a temporary anomalous condition where compact() is called on an UpdateSketch
- * that was initialized with p < 1.0 and update() was never called. In this case Theta < 1.0,
- * curCount = 0, and empty = true. The correction is to change Theta to 1.0, which makes the
- * returning sketch empty. This should only be used in the compaction or serialization of an
- * UpdateSketch.
- * @param empty the given empty state
- * @param curCount the given curCount
- * @param thetaLong the given thetaLong
- * @return thetaLong
- */ //This handles #4 above
- static final long correctThetaOnCompact(final boolean empty, final int curCount,
- final long thetaLong) {
- return (empty && (curCount == 0) && (thetaLong < Long.MAX_VALUE)) ? Long.MAX_VALUE : thetaLong;
- }
-
-
-
static final double estimate(final long thetaLong, final int curCount) {
return curCount * (LONG_MAX_VALUE_AS_DOUBLE / thetaLong);
}
diff --git a/src/main/java/org/apache/datasketches/theta/UpdateSketch.java b/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
index 5a14b6e..f94d8b7 100644
--- a/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
@@ -143,94 +143,6 @@ public abstract class UpdateSketch extends Sketch {
//return compact(this, dstOrdered, dstMem);
}
- static CompactSketch compact(final UpdateSketch usk, final boolean dstOrdered,
- final WritableMemory dstMem) {
- final int curCount = usk.getRetainedEntries(true);
- final boolean empty = usk.isEmpty();
- checkIllegalCurCountAndEmpty(empty, curCount);
-
- if (empty) {
- final EmptyCompactSketch sk = EmptyCompactSketch.getInstance();
- if (dstMem != null) {
- dstMem.putByteArray(0, sk.toByteArray(), 0, 8);
- }
- return sk;
- }
- //not empty
- final long thetaLong = correctThetaOnCompact(empty, curCount, usk.getThetaLong());
- final short seedHash = usk.getSeedHash();
- final long[] cache = usk.getCache();
- if ((thetaLong == Long.MAX_VALUE) && (curCount == 1)) {
-
- final long[] cacheOut = CompactOperations.compactCache(cache, curCount, thetaLong, dstOrdered);
- final long hash = cacheOut[0];
- final SingleItemSketch sis = new SingleItemSketch(hash, usk.getSeedHash());
- if (dstMem != null) {
- dstMem.putByteArray(0, sis.toByteArray(), 0, 16);
- }
- return sis;
- }
- if (dstMem == null) {
- final long[] cacheOut = CompactOperations.compactCache(cache, curCount, thetaLong, dstOrdered);
- if (dstOrdered) {
- return new HeapCompactOrderedSketch(cacheOut, false, seedHash, curCount, thetaLong);
- } else {
- return new HeapCompactUnorderedSketch(cacheOut, false, seedHash, curCount, thetaLong);
- }
- //return compactHeap(usk, dstOrdered, curCount, thetaLong);
- } else {
- return compactDirect(usk, dstMem, dstOrdered, curCount, thetaLong);
- }
- }
-
- /**
- * Converts the given UpdateSketch to a compact form.
- * EmptyCompactSketch and SingleItemSketch have already been checked.
- * @param sketch the given UpdateSketch
- * @param ordered true if the destination is to be ordered.
- * @param curCount the number of retained entries.
- * @param thetaLong the value of theta.
- * @return a CompactSketch
- */
- static CompactSketch compactHeap(final UpdateSketch sketch, final boolean ordered,
- final int curCount, final long thetaLong) {
- final short seedHash = sketch.getSeedHash();
- final long[] cache = sketch.getCache();
- final long[] cacheOut = CompactOperations.compactCache(cache, curCount, thetaLong, ordered);
- if (ordered) {
- return new HeapCompactOrderedSketch(cacheOut, false, seedHash, curCount, thetaLong);
- } else {
- return new HeapCompactUnorderedSketch(cacheOut, false, seedHash, curCount, thetaLong);
- }
- }
-
- /**
- * Converts the given UpdateSketch to a compact form.
- * EmptyCompactSketch and SingleItemSketch have already been checked.
- * @param sketch the given UpdateSketch
- * @param dstMem the given destination Memory. This clears it before use.
- * @param ordered true if the destination is to be ordered.
- * @param curCount the number of retained entries.
- * @param thetaLong the value of theta.
- * @return a CompactSketch.
- */
- static CompactSketch compactDirect(final UpdateSketch sketch,
- final WritableMemory dstMem, final boolean ordered, final int curCount, final long thetaLong) {
- final int preLongs = CompactOperations.computeCompactPreLongs(thetaLong, false, curCount);
- final short seedHash = sketch.getSeedHash();
- final long[] cache = sketch.getCache();
- final long[] cacheOut = CompactOperations.compactCache(cache, curCount, thetaLong, ordered);
- if (ordered) {
- final byte flags = (byte)(READ_ONLY_FLAG_MASK | COMPACT_FLAG_MASK | ORDERED_FLAG_MASK);
- CompactOperations.loadCompactMemory(cacheOut, seedHash, curCount, thetaLong, dstMem, flags, preLongs);
- } else {
- final byte flags = (byte)(READ_ONLY_FLAG_MASK | COMPACT_FLAG_MASK);
- CompactOperations.loadCompactMemory(cacheOut, seedHash, curCount, thetaLong, dstMem, flags, preLongs);
- }
- return new DirectCompactSketch(dstMem);
- }
-
-
@Override
public boolean isCompact() {
return false;
diff --git a/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java b/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
index a79f72a..a129766 100644
--- a/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
@@ -159,11 +159,7 @@ public class CompactSketchTest {
}
} else {
assertFalse(testSk.hasMemory());
- if (ordered) {
- assertTrue(testSk instanceof HeapCompactOrderedSketch);
- } else {
- assertTrue(testSk instanceof HeapCompactUnorderedSketch);
- }
+ assertTrue(testSk instanceof HeapCompactSketch);
}
assertEquals(testSk.getSeedHash(), refSk.getSeedHash());
assertEquals(testSk.getRetainedEntries(true), refSk.getRetainedEntries());
diff --git a/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java
index db69dd7..acca0b6 100644
--- a/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java
@@ -372,7 +372,7 @@ public class ConcurrentDirectQuickSelectSketchTest {
assertEquals(csk.getUpperBound(2), localUB);
assertFalse(csk.isEmpty());
assertEquals(csk.isEstimationMode(), estimating);
- assertEquals(csk.getClass().getSimpleName(), "HeapCompactUnorderedSketch");
+ assertEquals(csk.getClass().getSimpleName(), "HeapCompactSketch");
csk = shared.compact(true, null);
assertEquals(csk.getEstimate(), localEst);
@@ -380,7 +380,7 @@ public class ConcurrentDirectQuickSelectSketchTest {
assertEquals(csk.getUpperBound(2), localUB);
assertFalse(csk.isEmpty());
assertEquals(csk.isEstimationMode(), estimating);
- assertEquals(csk.getClass().getSimpleName(), "HeapCompactOrderedSketch");
+ assertEquals(csk.getClass().getSimpleName(), "HeapCompactSketch");
int bytes = local.getCurrentBytes(true);
assertEquals(bytes, (k*8) + (Family.COMPACT.getMaxPreLongs() << 3));
diff --git a/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java
index 21b580b..61bd5be 100644
--- a/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java
@@ -311,7 +311,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(comp1.isEmpty(), false);
assertEquals(comp1.isEstimationMode(), estimating);
assertEquals(comp1.getCurrentBytes(true), localCompBytes);
- assertEquals(comp1.getClass().getSimpleName(), "HeapCompactUnorderedSketch");
+ assertEquals(comp1.getClass().getSimpleName(), "HeapCompactSketch");
comp2 = shared.compact(true, null);
@@ -321,7 +321,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(comp2.isEmpty(), false);
assertEquals(comp2.isEstimationMode(), estimating);
assertEquals(comp2.getCurrentBytes(true), localCompBytes);
- assertEquals(comp2.getClass().getSimpleName(), "HeapCompactOrderedSketch");
+ assertEquals(comp2.getClass().getSimpleName(), "HeapCompactSketch");
byte[] memArr2 = new byte[localCompBytes];
WritableMemory mem2 = WritableMemory.wrap(memArr2); //allocate mem for compact form
diff --git a/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java
index 56070bc..d44625d 100644
--- a/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java
@@ -326,7 +326,7 @@ public class DirectQuickSelectSketchTest {
assertEquals(csk.getUpperBound(2), uskUB);
assertEquals(csk.isEmpty(), false);
assertEquals(csk.isEstimationMode(), estimating);
- assertEquals(csk.getClass().getSimpleName(), "HeapCompactUnorderedSketch");
+ assertEquals(csk.getClass().getSimpleName(), "HeapCompactSketch");
csk = usk.compact(true, null);
assertEquals(csk.getEstimate(), uskEst);
@@ -334,7 +334,7 @@ public class DirectQuickSelectSketchTest {
assertEquals(csk.getUpperBound(2), uskUB);
assertEquals(csk.isEmpty(), false);
assertEquals(csk.isEstimationMode(), estimating);
- assertEquals(csk.getClass().getSimpleName(), "HeapCompactOrderedSketch");
+ assertEquals(csk.getClass().getSimpleName(), "HeapCompactSketch");
int bytes = usk.getCurrentBytes(true);
assertEquals(bytes, (k*8) + (Family.COMPACT.getMaxPreLongs() << 3));
diff --git a/src/test/java/org/apache/datasketches/theta/ForwardCompatibilityTest.java b/src/test/java/org/apache/datasketches/theta/ForwardCompatibilityTest.java
index e5d040c..0a30d38 100644
--- a/src/test/java/org/apache/datasketches/theta/ForwardCompatibilityTest.java
+++ b/src/test/java/org/apache/datasketches/theta/ForwardCompatibilityTest.java
@@ -199,7 +199,7 @@ public class ForwardCompatibilityTest {
assertEquals(sketch.hasMemory(), false);
assertEquals(sketch.isCompact(), true);
assertEquals(sketch.isOrdered(), true);
- assertTrue(sketch instanceof HeapCompactOrderedSketch);
+ assertTrue(sketch instanceof HeapCompactSketch);
}
@Test
diff --git a/src/test/java/org/apache/datasketches/theta/HeapAlphaSketchTest.java b/src/test/java/org/apache/datasketches/theta/HeapAlphaSketchTest.java
index 12d300a..62ff8ef 100644
--- a/src/test/java/org/apache/datasketches/theta/HeapAlphaSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/HeapAlphaSketchTest.java
@@ -252,7 +252,7 @@ public class HeapAlphaSketchTest {
assertEquals(comp1.isEmpty(), false);
assertEquals(comp1.isEstimationMode(), estimating);
- assertEquals(comp1.getClass().getSimpleName(), "HeapCompactUnorderedSketch");
+ assertEquals(comp1.getClass().getSimpleName(), "HeapCompactSketch");
comp2 = usk.compact(true, null);
@@ -263,7 +263,7 @@ public class HeapAlphaSketchTest {
assertEquals(comp2.isEstimationMode(), estimating);
assertEquals(comp1bytes, comp2.getCurrentBytes(true)); //flag is not relevant
assertEquals(comp1curCount, comp2.getRetainedEntries(true)); //flag is not relevant
- assertEquals(comp2.getClass().getSimpleName(), "HeapCompactOrderedSketch");
+ assertEquals(comp2.getClass().getSimpleName(), "HeapCompactSketch");
int bytes = usk.getCurrentBytes(true);
int alphaBytes = sk1.getRetainedEntries(true) * 8;
diff --git a/src/test/java/org/apache/datasketches/theta/HeapQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/HeapQuickSelectSketchTest.java
index a75cb9a..c230a88 100644
--- a/src/test/java/org/apache/datasketches/theta/HeapQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/HeapQuickSelectSketchTest.java
@@ -251,7 +251,7 @@ public class HeapQuickSelectSketchTest {
assertEquals(comp1.isEmpty(), false);
assertEquals(comp1.isEstimationMode(), estimating);
assertEquals(comp1.getCurrentBytes(true), uskCompBytes);
- assertEquals(comp1.getClass().getSimpleName(), "HeapCompactUnorderedSketch");
+ assertEquals(comp1.getClass().getSimpleName(), "HeapCompactSketch");
comp2 = usk.compact(true, null);
@@ -261,7 +261,7 @@ public class HeapQuickSelectSketchTest {
assertEquals(comp2.isEmpty(), false);
assertEquals(comp2.isEstimationMode(), estimating);
assertEquals(comp2.getCurrentBytes(true), uskCompBytes);
- assertEquals(comp2.getClass().getSimpleName(), "HeapCompactOrderedSketch");
+ assertEquals(comp2.getClass().getSimpleName(), "HeapCompactSketch");
byte[] memArr2 = new byte[uskCompBytes];
WritableMemory mem2 = WritableMemory.wrap(memArr2); //allocate mem for compact form
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[incubator-datasketches-java] 02/10: Eliminated two Classes:
DirectCompactOrderedSketch and DirectCompactUnorderedSketch.
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leerho pushed a commit to branch Refactor_Theta_Tuple
in repository https://gitbox.apache.org/repos/asf/incubator-datasketches-java.git
commit f421f793d89e7543ced55960fd89738e28dfa98e
Author: Lee Rhodes <le...@users.noreply.github.com>
AuthorDate: Fri Jun 26 17:41:59 2020 -0700
Eliminated two Classes: DirectCompactOrderedSketch and
DirectCompactUnorderedSketch.
Replaced with just DirectCompactSketch.
---
.../datasketches/theta/CompactOperations.java | 17 ++----
.../theta/DirectCompactOrderedSketch.java | 60 ----------------------
.../datasketches/theta/DirectCompactSketch.java | 36 ++++++++++++-
.../theta/DirectCompactUnorderedSketch.java | 60 ----------------------
.../datasketches/theta/EmptyCompactSketch.java | 2 +-
.../datasketches/theta/SingleItemSketch.java | 2 +-
.../java/org/apache/datasketches/theta/Sketch.java | 5 +-
.../apache/datasketches/theta/UpdateSketch.java | 3 +-
.../datasketches/theta/CompactSketchTest.java | 24 ++++-----
.../ConcurrentDirectQuickSelectSketchTest.java | 8 +--
.../theta/ConcurrentHeapQuickSelectSketchTest.java | 12 ++---
.../theta/DirectQuickSelectSketchTest.java | 8 +--
.../datasketches/theta/HeapAlphaSketchTest.java | 8 +--
.../theta/HeapQuickSelectSketchTest.java | 8 +--
.../datasketches/theta/SingleItemSketchTest.java | 6 +--
.../org/apache/datasketches/theta/SketchTest.java | 6 ++-
.../apache/datasketches/theta/UnionImplTest.java | 6 +--
17 files changed, 87 insertions(+), 184 deletions(-)
diff --git a/src/main/java/org/apache/datasketches/theta/CompactOperations.java b/src/main/java/org/apache/datasketches/theta/CompactOperations.java
index 6c7067e..882e0e8 100644
--- a/src/main/java/org/apache/datasketches/theta/CompactOperations.java
+++ b/src/main/java/org/apache/datasketches/theta/CompactOperations.java
@@ -91,11 +91,8 @@ final class CompactOperations {
final Memory mem =
loadCompactMemory(hashArrOut, seedHash, curCount, thetaLong, dstMem, (byte)flags, preLongs);
- if (dstOrderedOut) {
- return new DirectCompactOrderedSketch(mem);
- } else {
- return new DirectCompactUnorderedSketch(mem);
- }
+ return new DirectCompactSketch(mem);
+
} else { //Heap
if (empty) {
return EmptyCompactSketch.getInstance();
@@ -159,7 +156,7 @@ final class CompactOperations {
if (srcEmptyFlag) {
if (dstMem != null) {
dstMem.putByteArray(0, EmptyCompactSketch.EMPTY_COMPACT_SKETCH_ARR, 0, 8);
- return new DirectCompactOrderedSketch(dstMem);
+ return new DirectCompactSketch(dstMem);
} else {
return EmptyCompactSketch.getInstance();
}
@@ -169,7 +166,7 @@ final class CompactOperations {
final SingleItemSketch sis = new SingleItemSketch(hash, srcSeedHash);
if (dstMem != null) {
dstMem.putByteArray(0, sis.toByteArray(),0, 16);
- return new DirectCompactOrderedSketch(dstMem);
+ return new DirectCompactSketch(dstMem);
} else { //heap
return sis;
}
@@ -191,11 +188,7 @@ final class CompactOperations {
if (dstMem != null) {
final Memory tgtMem = loadCompactMemory(hashArr, srcSeedHash, curCount, thetaLong, dstMem,
(byte)srcFlags, srcPreLongs);
- if (dstOrderedOut) {
- return new DirectCompactOrderedSketch(tgtMem);
- } else {
- return new DirectCompactUnorderedSketch(tgtMem);
- }
+ return new DirectCompactSketch(tgtMem);
} else { //heap
if (dstOrderedOut) {
diff --git a/src/main/java/org/apache/datasketches/theta/DirectCompactOrderedSketch.java b/src/main/java/org/apache/datasketches/theta/DirectCompactOrderedSketch.java
deleted file mode 100644
index e2c9de5..0000000
--- a/src/main/java/org/apache/datasketches/theta/DirectCompactOrderedSketch.java
+++ /dev/null
@@ -1,60 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.theta;
-
-import static org.apache.datasketches.theta.PreambleUtil.checkMemorySeedHash;
-
-import org.apache.datasketches.memory.Memory;
-
-/**
- * An off-heap (Direct), compact, ordered, read-only sketch. This sketch can only be associated
- * with a Serialization Version 3 format binary image.
- *
- * <p>This implementation uses data in a given Memory that is owned and managed by the caller.
- * This Memory can be off-heap, which if managed properly will greatly reduce the need for
- * the JVM to perform garbage collection.</p>
- *
- * @author Lee Rhodes
- */
-final class DirectCompactOrderedSketch extends DirectCompactSketch {
-
- DirectCompactOrderedSketch(final Memory mem) {
- super(mem);
- }
-
- /**
- * Wraps the given Memory, which must be a SerVer 3, ordered, Compact Sketch image.
- * Must check the validity of the Memory before calling.
- * @param srcMem <a href="{@docRoot}/resources/dictionary.html#mem">See Memory</a>
- * @param seed The update seed.
- * <a href="{@docRoot}/resources/dictionary.html#seed">See Update Hash Seed</a>.
- * @return this sketch
- */
- static DirectCompactOrderedSketch wrapInstance(final Memory srcMem, final long seed) {
- checkMemorySeedHash(srcMem, seed);
- return new DirectCompactOrderedSketch(srcMem);
- }
-
- @Override
- public boolean isOrdered() {
- return true;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java b/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java
index 6b65640..a695944 100644
--- a/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java
@@ -19,7 +19,10 @@
package org.apache.datasketches.theta;
+import static org.apache.datasketches.theta.PreambleUtil.ORDERED_FLAG_MASK;
+import static org.apache.datasketches.theta.PreambleUtil.checkMemorySeedHash;
import static org.apache.datasketches.theta.PreambleUtil.extractCurCount;
+import static org.apache.datasketches.theta.PreambleUtil.extractFlags;
import static org.apache.datasketches.theta.PreambleUtil.extractPreLongs;
import static org.apache.datasketches.theta.PreambleUtil.extractSeedHash;
import static org.apache.datasketches.theta.PreambleUtil.extractThetaLong;
@@ -29,17 +32,41 @@ import org.apache.datasketches.memory.Memory;
import org.apache.datasketches.memory.WritableMemory;
/**
- * Parent class of the Direct Compact Sketches.
+ * An off-heap (Direct), compact, read-only sketch. The internal hash array can be either ordered
+ * or unordered.
+ *
+ * <p>This sketch can only be associated with a Serialization Version 3 format binary image.</p>
+ *
+ * <p>This implementation uses data in a given Memory that is owned and managed by the caller.
+ * This Memory can be off-heap, which if managed properly will greatly reduce the need for
+ * the JVM to perform garbage collection.</p>
*
* @author Lee Rhodes
*/
-abstract class DirectCompactSketch extends CompactSketch {
+class DirectCompactSketch extends CompactSketch {
final Memory mem_;
+ /**
+ * Construct this sketch with the given memory.
+ * @param mem Read-only Memory object with the order bit properly set.
+ */
DirectCompactSketch(final Memory mem) {
mem_ = mem;
}
+ /**
+ * Wraps the given Memory, which must be a SerVer 3, ordered, CompactSketch image.
+ * Must check the validity of the Memory before calling. The order bit must be set properly.
+ * @param srcMem <a href="{@docRoot}/resources/dictionary.html#mem">See Memory</a>
+ * @param seed The update seed.
+ * <a href="{@docRoot}/resources/dictionary.html#seed">See Update Hash Seed</a>.
+ * @return this sketch
+ */
+ static DirectCompactSketch wrapInstance(final Memory srcMem, final long seed) {
+ checkMemorySeedHash(srcMem, seed);
+ return new DirectCompactSketch(srcMem);
+ }
+
//Sketch
@Override
@@ -102,6 +129,11 @@ abstract class DirectCompactSketch extends CompactSketch {
}
@Override
+ public boolean isOrdered() {
+ return (extractFlags(mem_) & ORDERED_FLAG_MASK) > 0;
+ }
+
+ @Override
public boolean isSameResource(final Memory that) {
return mem_.isSameResource(that);
}
diff --git a/src/main/java/org/apache/datasketches/theta/DirectCompactUnorderedSketch.java b/src/main/java/org/apache/datasketches/theta/DirectCompactUnorderedSketch.java
deleted file mode 100644
index 15b4433..0000000
--- a/src/main/java/org/apache/datasketches/theta/DirectCompactUnorderedSketch.java
+++ /dev/null
@@ -1,60 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.theta;
-
-import static org.apache.datasketches.theta.PreambleUtil.checkMemorySeedHash;
-
-import org.apache.datasketches.memory.Memory;
-
-/**
- * An off-heap (Direct), compact, unordered, read-only sketch. This sketch can only be associated
- * with a Serialization Version 3 format binary image.
- *
- * <p>This implementation uses data in a given Memory that is owned and managed by the caller.
- * This Memory can be off-heap, which if managed properly will greatly reduce the need for
- * the JVM to perform garbage collection.</p>
- *
- * @author Lee Rhodes
- */
-final class DirectCompactUnorderedSketch extends DirectCompactSketch {
-
- DirectCompactUnorderedSketch(final Memory mem) {
- super(mem);
- }
-
- /**
- * Wraps the given Memory, which must be a SerVer 3, unordered, Compact Sketch image.
- * Must check the validity of the Memory before calling.
- * @param srcMem <a href="{@docRoot}/resources/dictionary.html#mem">See Memory</a>
- * @param seed The update seed.
- * <a href="{@docRoot}/resources/dictionary.html#seed">See Update Hash Seed</a>.
- * @return this sketch
- */
- static DirectCompactUnorderedSketch wrapInstance(final Memory srcMem, final long seed) {
- checkMemorySeedHash(srcMem, seed);
- return new DirectCompactUnorderedSketch(srcMem);
- }
-
- @Override
- public boolean isOrdered() {
- return false;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/theta/EmptyCompactSketch.java b/src/main/java/org/apache/datasketches/theta/EmptyCompactSketch.java
index 666a9cd..9993673 100644
--- a/src/main/java/org/apache/datasketches/theta/EmptyCompactSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/EmptyCompactSketch.java
@@ -70,7 +70,7 @@ final class EmptyCompactSketch extends CompactSketch {
// The hashSeed is ignored and set == 0.
public CompactSketch compact(final boolean dstOrdered, final WritableMemory wmem) {
wmem.putByteArray(0, EMPTY_COMPACT_SKETCH_ARR, 0, 8);
- return new DirectCompactOrderedSketch(wmem);
+ return new DirectCompactSketch(wmem);
}
//static
diff --git a/src/main/java/org/apache/datasketches/theta/SingleItemSketch.java b/src/main/java/org/apache/datasketches/theta/SingleItemSketch.java
index 5356dbc..512c81c 100644
--- a/src/main/java/org/apache/datasketches/theta/SingleItemSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/SingleItemSketch.java
@@ -110,7 +110,7 @@ final class SingleItemSketch extends CompactSketch {
else {
dstMem.putLong(0, pre0_);
dstMem.putLong(8, hash_);
- return new DirectCompactOrderedSketch(dstMem);
+ return new DirectCompactSketch(dstMem);
}
}
diff --git a/src/main/java/org/apache/datasketches/theta/Sketch.java b/src/main/java/org/apache/datasketches/theta/Sketch.java
index 42f4724..b8c7bd7 100644
--- a/src/main/java/org/apache/datasketches/theta/Sketch.java
+++ b/src/main/java/org/apache/datasketches/theta/Sketch.java
@@ -153,7 +153,6 @@ public abstract class Sketch {
}
//not empty & not singleItem
final int flags = srcMem.getByte(FLAGS_BYTE);
- final boolean orderedFlag = (flags & ORDERED_FLAG_MASK) > 0;
final boolean compactFlag = (flags & COMPACT_FLAG_MASK) > 0;
if (!compactFlag) {
throw new SketchesArgumentException(
@@ -164,9 +163,7 @@ public abstract class Sketch {
throw new SketchesArgumentException(
"Corrupted: COMPACT family sketch image must have Read-Only flag set");
}
- return orderedFlag
- ? DirectCompactOrderedSketch.wrapInstance(srcMem, seed)
- : DirectCompactUnorderedSketch.wrapInstance(srcMem, seed);
+ return DirectCompactSketch.wrapInstance(srcMem, seed);
} //end of serVer 3
else if (serVer == 1) {
return ForwardCompatibility.heapify1to3(srcMem, seed);
diff --git a/src/main/java/org/apache/datasketches/theta/UpdateSketch.java b/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
index 686789d..5a14b6e 100644
--- a/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
@@ -223,12 +223,11 @@ public abstract class UpdateSketch extends Sketch {
if (ordered) {
final byte flags = (byte)(READ_ONLY_FLAG_MASK | COMPACT_FLAG_MASK | ORDERED_FLAG_MASK);
CompactOperations.loadCompactMemory(cacheOut, seedHash, curCount, thetaLong, dstMem, flags, preLongs);
- return new DirectCompactOrderedSketch(dstMem);
} else {
final byte flags = (byte)(READ_ONLY_FLAG_MASK | COMPACT_FLAG_MASK);
CompactOperations.loadCompactMemory(cacheOut, seedHash, curCount, thetaLong, dstMem, flags, preLongs);
- return new DirectCompactUnorderedSketch(dstMem);
}
+ return new DirectCompactSketch(dstMem);
}
diff --git a/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java b/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
index 199043f..a79f72a 100644
--- a/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
@@ -148,9 +148,9 @@ public class CompactSketchTest {
assertTrue(testSk.hasMemory());
assertNotNull(testSk.getMemory());
if (ordered) {
- assertTrue(testSk instanceof DirectCompactOrderedSketch);
+ assertTrue(testSk.isOrdered());
} else {
- assertTrue(testSk instanceof DirectCompactUnorderedSketch);
+ assertFalse(testSk.isOrdered());
}
if (refSk.isDirect()) {
assertTrue(testSk.isDirect());
@@ -237,23 +237,23 @@ public class CompactSketchTest {
WritableMemory wmem = WritableMemory.allocate(16);
csko = sk.compact(true, wmem); //empty, direct, ordered
//ClassType, Count, Bytes, Compact, Empty, Direct, Memory, Ordered, Estimation
- state = new State("DirectCompactOrderedSketch", 0, 8, true, true, false, true, true, false);
+ state = new State("DirectCompactSketch", 0, 8, true, true, false, true, true, false);
state.check(csko);
wmem = WritableMemory.allocate(16);
csku = sk.compact(false, wmem); //empty, direct, unordered
- state = new State("DirectCompactOrderedSketch", 0, 8, true, true, false, true, true, false);
+ state = new State("DirectCompactSketch", 0, 8, true, true, false, true, true, false);
state.check(csku);
sk.update(1);
wmem = WritableMemory.allocate(16);
csko = sk.compact(true, wmem); //Single, direct, ordered
- state = new State("DirectCompactOrderedSketch", 1, 16, true, false, false, true, true, false);
+ state = new State("DirectCompactSketch", 1, 16, true, false, false, true, true, false);
state.check(csko);
wmem = WritableMemory.allocate(16);
csku = sk.compact(false, wmem); //Single, direct, unordered
- state = new State("DirectCompactOrderedSketch", 1, 16, true, false, false, true, true, false);
+ state = new State("DirectCompactSketch", 1, 16, true, false, false, true, true, false);
state.check(csku);
CompactSketch csk2o; //ordered
@@ -277,24 +277,24 @@ public class CompactSketchTest {
wmem = WritableMemory.allocate(16);
csk2o = csku.compact(true, wmem);
- state.classType = "DirectCompactOrderedSketch";
+ state.classType = "DirectCompactSketch";
state.memory = true;
state.check(csk2o);
wmem = WritableMemory.allocate(16);
csk2u = csku.compact(false, wmem);
- state.classType = "DirectCompactOrderedSketch";
+ state.classType = "DirectCompactSketch";
state.check(csk2u);
wmem = WritableMemory.allocate(16);
csk2o = csko.compact(true, wmem);
- state.classType = "DirectCompactOrderedSketch";
+ state.classType = "DirectCompactSketch";
state.memory = true;
state.check(csk2o);
wmem = WritableMemory.allocate(16);
csk2u = csko.compact(false, wmem);
- state.classType = "DirectCompactOrderedSketch";
+ state.classType = "DirectCompactSketch";
state.check(csk2u);
}
@@ -313,8 +313,8 @@ public class CompactSketchTest {
public void checkHeapifyEmptySketch() {
UpdateSketch sk = Sketches.updateSketchBuilder().build();
WritableMemory wmem = WritableMemory.allocate(16); //empty, but extra bytes
- CompactSketch csk = sk.compact(false, wmem);
- assertTrue(csk instanceof DirectCompactOrderedSketch);
+ CompactSketch csk = sk.compact(false, wmem); //ignores order because it is empty
+ assertTrue(csk instanceof DirectCompactSketch);
Sketch csk2 = Sketch.heapify(wmem);
assertTrue(csk2 instanceof EmptyCompactSketch);
}
diff --git a/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java
index 3802dfd..db69dd7 100644
--- a/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java
@@ -393,7 +393,7 @@ public class ConcurrentDirectQuickSelectSketchTest {
assertEquals(csk.getUpperBound(2), localUB);
assertFalse(csk.isEmpty());
assertEquals(csk.isEstimationMode(), estimating);
- assertEquals(csk.getClass().getSimpleName(), "DirectCompactUnorderedSketch");
+ assertEquals(csk.getClass().getSimpleName(), "DirectCompactSketch");
mem2.clear();
csk = shared.compact(true, mem2);
@@ -402,7 +402,7 @@ public class ConcurrentDirectQuickSelectSketchTest {
assertEquals(csk.getUpperBound(2), localUB);
assertFalse(csk.isEmpty());
assertEquals(csk.isEstimationMode(), estimating);
- assertEquals(csk.getClass().getSimpleName(), "DirectCompactOrderedSketch");
+ assertEquals(csk.getClass().getSimpleName(), "DirectCompactSketch");
csk.toString(false, true, 0, false);
}
}
@@ -438,7 +438,7 @@ public class ConcurrentDirectQuickSelectSketchTest {
assertEquals(csk2.getUpperBound(2), localUB);
assertTrue(csk2.isEmpty());
assertFalse(csk2.isEstimationMode());
- assertTrue(csk2 instanceof DirectCompactOrderedSketch);
+ assertTrue(csk2.isOrdered());
CompactSketch csk3 = shared.compact(true, mem2);
csk3.toString(false, true, 0, false);
csk3.toString();
@@ -447,7 +447,7 @@ public class ConcurrentDirectQuickSelectSketchTest {
assertEquals(csk3.getUpperBound(2), localUB);
assertTrue(csk3.isEmpty());
assertFalse(csk3.isEstimationMode());
- assertTrue(csk2 instanceof DirectCompactOrderedSketch);
+ assertTrue(csk2.isOrdered());
}
}
diff --git a/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java
index 8ae2d89..21b580b 100644
--- a/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java
@@ -334,7 +334,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(comp3.isEmpty(), false);
assertEquals(comp3.isEstimationMode(), estimating);
assertEquals(comp3.getCurrentBytes(true), localCompBytes);
- assertEquals(comp3.getClass().getSimpleName(), "DirectCompactUnorderedSketch");
+ assertEquals(comp3.getClass().getSimpleName(), "DirectCompactSketch");
mem2.clear();
comp4 = shared.compact(true, mem2);
@@ -345,7 +345,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(comp4.isEmpty(), false);
assertEquals(comp4.isEstimationMode(), estimating);
assertEquals(comp4.getCurrentBytes(true), localCompBytes);
- assertEquals(comp4.getClass().getSimpleName(), "DirectCompactOrderedSketch");
+ assertEquals(comp4.getClass().getSimpleName(), "DirectCompactSketch");
comp4.toString(false, true, 0, false);
}
@@ -381,7 +381,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(csk2.getUpperBound(2), localUB);
assertEquals(csk2.isEmpty(), true);
assertEquals(csk2.isEstimationMode(), estimating);
- assertTrue(csk2 instanceof DirectCompactOrderedSketch);
+ assertTrue(csk2.isOrdered());
CompactSketch csk3 = shared.compact(true, mem2);
csk3.toString(false, true, 0, false);
@@ -391,7 +391,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(csk3.getUpperBound(2), localUB);
assertEquals(csk3.isEmpty(), true);
assertEquals(csk3.isEstimationMode(), estimating);
- assertTrue(csk3 instanceof DirectCompactOrderedSketch);
+ assertTrue(csk3.isOrdered());
}
@Test
@@ -593,7 +593,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(csk2.getUpperBound(2), uskUB);
assertTrue(csk2.isEmpty());
assertFalse(csk2.isEstimationMode());
- assertTrue(csk2 instanceof DirectCompactOrderedSketch);
+ assertTrue(csk2.isOrdered());
CompactSketch csk3 = shared.compact(true, mem2);
csk3.toString(false, true, 0, false);
@@ -603,7 +603,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(csk3.getUpperBound(2), uskUB);
assertTrue(csk3.isEmpty());
assertFalse(csk3.isEstimationMode());
- assertTrue(csk2 instanceof DirectCompactOrderedSketch);
+ assertTrue(csk2.isOrdered());
}
@Test(expectedExceptions = SketchesArgumentException.class)
diff --git a/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java
index 73245e5..56070bc 100644
--- a/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java
@@ -347,7 +347,7 @@ public class DirectQuickSelectSketchTest {
assertEquals(csk.getUpperBound(2), uskUB);
assertEquals(csk.isEmpty(), false);
assertEquals(csk.isEstimationMode(), estimating);
- assertEquals(csk.getClass().getSimpleName(), "DirectCompactUnorderedSketch");
+ assertEquals(csk.getClass().getSimpleName(), "DirectCompactSketch");
mem2.clear();
csk = usk.compact(true, mem2);
@@ -356,7 +356,7 @@ public class DirectQuickSelectSketchTest {
assertEquals(csk.getUpperBound(2), uskUB);
assertEquals(csk.isEmpty(), false);
assertEquals(csk.isEstimationMode(), estimating);
- assertEquals(csk.getClass().getSimpleName(), "DirectCompactOrderedSketch");
+ assertEquals(csk.getClass().getSimpleName(), "DirectCompactSketch");
csk.toString(false, true, 0, false);
}
}
@@ -388,7 +388,7 @@ public class DirectQuickSelectSketchTest {
assertEquals(csk2.getUpperBound(2), uskUB);
assertEquals(csk2.isEmpty(), true);
assertEquals(csk2.isEstimationMode(), false);
- assertEquals(csk2.getClass().getSimpleName(), "DirectCompactOrderedSketch");
+ assertEquals(csk2.getClass().getSimpleName(), "DirectCompactSketch");
CompactSketch csk3 = usk.compact(true, mem2);
csk3.toString(false, true, 0, false);
@@ -398,7 +398,7 @@ public class DirectQuickSelectSketchTest {
assertEquals(csk3.getUpperBound(2), uskUB);
assertEquals(csk3.isEmpty(), true);
assertEquals(csk3.isEstimationMode(), false);
- assertEquals(csk3.getClass().getSimpleName(), "DirectCompactOrderedSketch");
+ assertEquals(csk3.getClass().getSimpleName(), "DirectCompactSketch");
}
}
diff --git a/src/test/java/org/apache/datasketches/theta/HeapAlphaSketchTest.java b/src/test/java/org/apache/datasketches/theta/HeapAlphaSketchTest.java
index 793eb13..12d300a 100644
--- a/src/test/java/org/apache/datasketches/theta/HeapAlphaSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/HeapAlphaSketchTest.java
@@ -280,7 +280,7 @@ public class HeapAlphaSketchTest {
assertEquals(comp3.isEstimationMode(), estimating);
assertEquals(comp1bytes, comp3.getCurrentBytes(true)); //flag is not relevant
assertEquals(comp1curCount, comp3.getRetainedEntries(true)); //flag is not relevant
- assertEquals(comp3.getClass().getSimpleName(), "DirectCompactUnorderedSketch");
+ assertEquals(comp3.getClass().getSimpleName(), "DirectCompactSketch");
mem2.clear();
comp4 = usk.compact(true, mem2);
@@ -292,7 +292,7 @@ public class HeapAlphaSketchTest {
assertEquals(comp4.isEstimationMode(), estimating);
assertEquals(comp1bytes, comp4.getCurrentBytes(true)); //flag is not relevant
assertEquals(comp1curCount, comp4.getRetainedEntries(true)); //flag is not relevant
- assertEquals(comp4.getClass().getSimpleName(), "DirectCompactOrderedSketch");
+ assertEquals(comp4.getClass().getSimpleName(), "DirectCompactSketch");
}
@Test
@@ -321,7 +321,7 @@ public class HeapAlphaSketchTest {
assertEquals(csk2.getUpperBound(2), uskUB);
assertEquals(csk2.isEmpty(), true);
assertEquals(csk2.isEstimationMode(), estimating);
- assertTrue(csk2 instanceof DirectCompactOrderedSketch);
+ assertTrue(csk2.isOrdered());
CompactSketch csk3 = usk.compact(true, mem2);
csk3.toString(false, true, 0, false);
@@ -331,7 +331,7 @@ public class HeapAlphaSketchTest {
assertEquals(csk3.getUpperBound(2), uskUB);
assertEquals(csk3.isEmpty(), true);
assertEquals(csk3.isEstimationMode(), estimating);
- assertTrue(csk3 instanceof DirectCompactOrderedSketch);
+ assertTrue(csk3.isOrdered());
}
@Test
diff --git a/src/test/java/org/apache/datasketches/theta/HeapQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/HeapQuickSelectSketchTest.java
index fcec52b..a75cb9a 100644
--- a/src/test/java/org/apache/datasketches/theta/HeapQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/HeapQuickSelectSketchTest.java
@@ -274,7 +274,7 @@ public class HeapQuickSelectSketchTest {
assertEquals(comp3.isEmpty(), false);
assertEquals(comp3.isEstimationMode(), estimating);
assertEquals(comp3.getCurrentBytes(true), uskCompBytes);
- assertEquals(comp3.getClass().getSimpleName(), "DirectCompactUnorderedSketch");
+ assertEquals(comp3.getClass().getSimpleName(), "DirectCompactSketch");
mem2.clear();
comp4 = usk.compact(true, mem2);
@@ -285,7 +285,7 @@ public class HeapQuickSelectSketchTest {
assertEquals(comp4.isEmpty(), false);
assertEquals(comp4.isEstimationMode(), estimating);
assertEquals(comp4.getCurrentBytes(true), uskCompBytes);
- assertEquals(comp4.getClass().getSimpleName(), "DirectCompactOrderedSketch");
+ assertEquals(comp4.getClass().getSimpleName(), "DirectCompactSketch");
comp4.toString(false, true, 0, false);
}
@@ -318,7 +318,7 @@ public class HeapQuickSelectSketchTest {
assertEquals(csk2.getUpperBound(2), uskUB);
assertEquals(csk2.isEmpty(), true);
assertEquals(csk2.isEstimationMode(), estimating);
- assertEquals(csk2.getClass().getSimpleName(), "DirectCompactOrderedSketch");
+ assertEquals(csk2.getClass().getSimpleName(), "DirectCompactSketch");
CompactSketch csk3 = usk.compact(true, mem2);
println(csk3.toString(false, true, 0, false));
@@ -328,7 +328,7 @@ public class HeapQuickSelectSketchTest {
assertEquals(csk3.getUpperBound(2), uskUB);
assertEquals(csk3.isEmpty(), true);
assertEquals(csk3.isEstimationMode(), estimating);
- assertEquals(csk3.getClass().getSimpleName(), "DirectCompactOrderedSketch");
+ assertEquals(csk3.getClass().getSimpleName(), "DirectCompactSketch");
}
@Test
diff --git a/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java b/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java
index 7a50ba8..dbbc6ff 100644
--- a/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java
@@ -205,9 +205,9 @@ public class SingleItemSketchTest {
bytes = Sketches.getMaxCompactSketchBytes(1);
wmem = WritableMemory.wrap(new byte[bytes]);
csk = sk1.compact(true, wmem);
- assertTrue(csk instanceof DirectCompactOrderedSketch);
+ assertTrue(csk.isOrdered());
csk = sk1.compact(false, wmem);
- assertTrue(csk instanceof DirectCompactOrderedSketch);
+ assertTrue(csk.isOrdered());
}
@Test
@@ -294,7 +294,7 @@ public class SingleItemSketchTest {
inter.update(sk2);
WritableMemory wmem = WritableMemory.wrap(new byte[16]);
CompactSketch csk = inter.getResult(false, wmem);
- assertTrue(csk instanceof DirectCompactOrderedSketch);
+ assertTrue(csk.isOrdered());
Sketch csk2 = Sketches.heapifySketch(wmem);
assertTrue(csk2 instanceof SingleItemSketch);
println(csk2.toString(true, true, 1, true));
diff --git a/src/test/java/org/apache/datasketches/theta/SketchTest.java b/src/test/java/org/apache/datasketches/theta/SketchTest.java
index 037dd47..db8ac34 100644
--- a/src/test/java/org/apache/datasketches/theta/SketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/SketchTest.java
@@ -329,11 +329,13 @@ public class SketchTest {
sketch.update(1);
sketch.update(2);
assertTrue(sketch.isSameResource(mem));
- DirectCompactOrderedSketch dcos = (DirectCompactOrderedSketch) sketch.compact(true, cmem);
+ DirectCompactSketch dcos = (DirectCompactSketch) sketch.compact(true, cmem);
assertTrue(dcos.isSameResource(cmem));
+ assertTrue(dcos.isOrdered());
//never create 2 sketches with the same memory, so don't do as I do :)
- DirectCompactUnorderedSketch dcs = (DirectCompactUnorderedSketch) sketch.compact(false, cmem);
+ DirectCompactSketch dcs = (DirectCompactSketch) sketch.compact(false, cmem);
assertTrue(dcs.isSameResource(cmem));
+ assertFalse(dcs.isOrdered());
Sketch sk = Sketches.updateSketchBuilder().setNominalEntries(k).build();
assertFalse(sk.isSameResource(mem));
diff --git a/src/test/java/org/apache/datasketches/theta/UnionImplTest.java b/src/test/java/org/apache/datasketches/theta/UnionImplTest.java
index b1d7a7f..c7d2b50 100644
--- a/src/test/java/org/apache/datasketches/theta/UnionImplTest.java
+++ b/src/test/java/org/apache/datasketches/theta/UnionImplTest.java
@@ -54,14 +54,14 @@ public class UnionImplTest {
assertEquals(union.getResult().getEstimate(), k, 0.0);
}
- @Test(expectedExceptions = SketchesArgumentException.class)
- public void checkCorruptedCompactFlag() {
+ @Test
+ public void checkUnorderedCompactFlag() {
int k = 16;
WritableMemory mem = WritableMemory.wrap(new byte[(k*8) + 24]);
UpdateSketch sketch = Sketches.updateSketchBuilder().setNominalEntries(k).build();
for (int i=0; i<k; i++) { sketch.update(i); }
CompactSketch sketchInDirectOrd = sketch.compact(true, mem);
- sketch.compact(false, mem); //corrupt memory
+ sketch.compact(false, mem); //change the order bit
Union union = Sketches.setOperationBuilder().setNominalEntries(k).buildUnion();
union.update(sketchInDirectOrd);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[incubator-datasketches-java] 04/10: Interim with errors
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leerho pushed a commit to branch Refactor_Theta_Tuple
in repository https://gitbox.apache.org/repos/asf/incubator-datasketches-java.git
commit 184623eb04b88689cf9012c22765cde3d84caab3
Author: Lee Rhodes <le...@users.noreply.github.com>
AuthorDate: Mon Jun 29 08:48:09 2020 -0700
Interim with errors
---
.../datasketches/theta/CompactOperations.java | 15 ++++++++--
.../apache/datasketches/theta/CompactSketch.java | 10 +++++++
.../theta/ConcurrentHeapThetaBuffer.java | 4 +--
.../theta/ConcurrentSharedThetaSketch.java | 4 ++-
.../datasketches/theta/DirectCompactSketch.java | 2 +-
.../theta/DirectQuickSelectSketchR.java | 33 ++++++++++------------
.../datasketches/theta/EmptyCompactSketch.java | 2 +-
.../apache/datasketches/theta/HeapAlphaSketch.java | 2 +-
.../datasketches/theta/HeapCompactSketch.java | 6 ++--
.../datasketches/theta/HeapQuickSelectSketch.java | 2 +-
.../datasketches/theta/HeapUpdateSketch.java | 21 ++++++++++----
.../datasketches/theta/SingleItemSketch.java | 2 +-
.../java/org/apache/datasketches/theta/Sketch.java | 15 ++++++++--
.../apache/datasketches/theta/UpdateSketch.java | 12 +++++++-
.../apache/datasketches/theta/AnotBimplTest.java | 8 +++---
.../datasketches/theta/CompactSketchTest.java | 18 ++++++------
.../ConcurrentDirectQuickSelectSketchTest.java | 4 +--
.../org/apache/datasketches/theta/SketchTest.java | 23 ++++++++-------
18 files changed, 114 insertions(+), 69 deletions(-)
diff --git a/src/main/java/org/apache/datasketches/theta/CompactOperations.java b/src/main/java/org/apache/datasketches/theta/CompactOperations.java
index 2c17ade..2f07079 100644
--- a/src/main/java/org/apache/datasketches/theta/CompactOperations.java
+++ b/src/main/java/org/apache/datasketches/theta/CompactOperations.java
@@ -83,7 +83,7 @@ final class CompactOperations {
//Note: for empty and single we always output the ordered form.
final boolean dstOrderedOut = (empty || single) ? true : dstOrdered;
if (direct) {
- final int preLongs = computeCompactPreLongs(thetaLong, empty, curCount);
+ final int preLongs = computeCompactPreLongs(empty, curCount, thetaLong);
int flags = READ_ONLY_FLAG_MASK | COMPACT_FLAG_MASK; //always LE
flags |= empty ? EMPTY_FLAG_MASK : 0;
flags |= dstOrderedOut ? ORDERED_FLAG_MASK : 0;
@@ -341,8 +341,17 @@ final class CompactOperations {
}
}
- static final int computeCompactPreLongs(final long thetaLong, final boolean empty,
- final int curCount) {
+ /**
+ * This compute number of preamble longs for a compact sketch based on <i>empty</i>,
+ * <i>curCount</i> and <i>thetaLong</i>.
+ * This also accommodates for EmptyCompactSketch and SingleItemSketch.
+ * @param empty The given empty state
+ * @param curCount The given current count (retained entries)
+ * @param thetaLong the current thetaLong
+ * @return the number of preamble longs
+ */
+ static final int computeCompactPreLongs(final boolean empty, final int curCount,
+ final long thetaLong) {
return (thetaLong < Long.MAX_VALUE) ? 3 : empty ? 1 : (curCount > 1) ? 2 : 1;
}
diff --git a/src/main/java/org/apache/datasketches/theta/CompactSketch.java b/src/main/java/org/apache/datasketches/theta/CompactSketch.java
index faafd5c..e99ecf3 100644
--- a/src/main/java/org/apache/datasketches/theta/CompactSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/CompactSketch.java
@@ -46,6 +46,16 @@ public abstract class CompactSketch extends Sketch {
public abstract CompactSketch compact(final boolean dstOrdered, final WritableMemory dstMem);
@Override
+ public int getCompactBytes() {
+ return getCurrentBytes();
+ }
+
+ @Override
+ int getCurrentDataLongs() {
+ return getRetainedEntries(true);
+ }
+
+ @Override
public Family getFamily() {
return Family.COMPACT;
}
diff --git a/src/main/java/org/apache/datasketches/theta/ConcurrentHeapThetaBuffer.java b/src/main/java/org/apache/datasketches/theta/ConcurrentHeapThetaBuffer.java
index f17e89b..64ce463 100644
--- a/src/main/java/org/apache/datasketches/theta/ConcurrentHeapThetaBuffer.java
+++ b/src/main/java/org/apache/datasketches/theta/ConcurrentHeapThetaBuffer.java
@@ -78,8 +78,8 @@ final class ConcurrentHeapThetaBuffer extends HeapQuickSelectSketch {
//Sketch overrides
@Override
- public int getCurrentBytes(final boolean compact) {
- return shared.getCurrentBytes(compact);
+ public int getCurrentBytes() {
+ return shared.getCurrentBytes();
}
@Override
diff --git a/src/main/java/org/apache/datasketches/theta/ConcurrentSharedThetaSketch.java b/src/main/java/org/apache/datasketches/theta/ConcurrentSharedThetaSketch.java
index fc5a00a..3c2c4b1 100644
--- a/src/main/java/org/apache/datasketches/theta/ConcurrentSharedThetaSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/ConcurrentSharedThetaSketch.java
@@ -118,7 +118,9 @@ interface ConcurrentSharedThetaSketch {
// inheritance. They are provided here to allow casts to this interface access
// to these methods without having to cast back to the extended parent class.
- int getCurrentBytes(boolean compact);
+ int getCompactBytes();
+
+ int getCurrentBytes();
double getEstimate();
diff --git a/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java b/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java
index aac4431..458a93d 100644
--- a/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java
@@ -82,7 +82,7 @@ class DirectCompactSketch extends CompactSketch {
}
@Override
- public int getCurrentBytes(final boolean compact) { //compact is ignored here
+ public int getCurrentBytes() {
if (otherCheckForSingleItem(mem_)) { return 16; }
final int preLongs = extractPreLongs(mem_);
final int curCount = (preLongs == 1) ? 0 : extractCurCount(mem_);
diff --git a/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java b/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java
index f5bfb49..36f4da3 100644
--- a/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java
+++ b/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java
@@ -20,6 +20,9 @@
package org.apache.datasketches.theta;
import static org.apache.datasketches.Util.REBUILD_THRESHOLD;
+import static org.apache.datasketches.theta.CompactOperations.checkIllegalCurCountAndEmpty;
+import static org.apache.datasketches.theta.CompactOperations.computeCompactPreLongs;
+import static org.apache.datasketches.theta.CompactOperations.correctThetaOnCompact;
import static org.apache.datasketches.theta.PreambleUtil.FAMILY_BYTE;
import static org.apache.datasketches.theta.PreambleUtil.LG_ARR_LONGS_BYTE;
import static org.apache.datasketches.theta.PreambleUtil.LG_NOM_LONGS_BYTE;
@@ -33,6 +36,7 @@ import static org.apache.datasketches.theta.PreambleUtil.extractLgArrLongs;
import static org.apache.datasketches.theta.PreambleUtil.extractLgNomLongs;
import static org.apache.datasketches.theta.PreambleUtil.extractPreLongs;
import static org.apache.datasketches.theta.PreambleUtil.extractThetaLong;
+import static org.apache.datasketches.theta.PreambleUtil.insertThetaLong;
import org.apache.datasketches.Family;
import org.apache.datasketches.ResizeFactor;
@@ -107,17 +111,12 @@ class DirectQuickSelectSketchR extends UpdateSketch {
//Sketch
@Override
- public int getCurrentBytes(final boolean compact) {
- if (!compact) {
- final byte lgArrLongs = wmem_.getByte(LG_ARR_LONGS_BYTE);
- final int preambleLongs = wmem_.getByte(PREAMBLE_LONGS_BYTE) & 0X3F;
- final int lengthBytes = (preambleLongs + (1 << lgArrLongs)) << 3;
- return lengthBytes;
- }
- final int preLongs = getCurrentPreambleLongs(true);
- final int curCount = getRetainedEntries(true);
-
- return (preLongs + curCount) << 3;
+ public int getCurrentBytes() {
+ //not compact
+ final byte lgArrLongs = wmem_.getByte(LG_ARR_LONGS_BYTE);
+ final int preLongs = wmem_.getByte(PREAMBLE_LONGS_BYTE) & 0X3F;
+ final int lengthBytes = (preLongs + (1 << lgArrLongs)) << 3;
+ return lengthBytes;
}
@Override
@@ -170,16 +169,14 @@ class DirectQuickSelectSketchR extends UpdateSketch {
@Override
public byte[] toByteArray() { //MY_FAMILY is stored in wmem_
- CompactOperations.checkIllegalCurCountAndEmpty(isEmpty(), extractCurCount(wmem_));
- final byte lgArrLongs = wmem_.getByte(LG_ARR_LONGS_BYTE);
- final int preambleLongs = wmem_.getByte(PREAMBLE_LONGS_BYTE) & 0X3F;
- final int lengthBytes = (preambleLongs + (1 << lgArrLongs)) << 3;
+ checkIllegalCurCountAndEmpty(isEmpty(), extractCurCount(wmem_));
+ final int lengthBytes = getCurrentBytes();
final byte[] byteArray = new byte[lengthBytes];
final WritableMemory mem = WritableMemory.wrap(byteArray);
wmem_.copyTo(0, mem, 0, lengthBytes);
final long thetaLong =
- CompactOperations.correctThetaOnCompact(isEmpty(), extractCurCount(wmem_), extractThetaLong(wmem_));
- PreambleUtil.insertThetaLong(wmem_, thetaLong);
+ correctThetaOnCompact(isEmpty(), extractCurCount(wmem_), extractThetaLong(wmem_));
+ insertThetaLong(wmem_, thetaLong);
return byteArray;
}
@@ -220,7 +217,7 @@ class DirectQuickSelectSketchR extends UpdateSketch {
@Override
int getCurrentPreambleLongs(final boolean compact) {
if (!compact) { return PreambleUtil.extractPreLongs(wmem_); }
- return CompactOperations.computeCompactPreLongs(getThetaLong(), isEmpty(), getRetainedEntries(true));
+ return computeCompactPreLongs(isEmpty(), getRetainedEntries(true), getThetaLong());
}
@Override
diff --git a/src/main/java/org/apache/datasketches/theta/EmptyCompactSketch.java b/src/main/java/org/apache/datasketches/theta/EmptyCompactSketch.java
index 9993673..54c71eb 100644
--- a/src/main/java/org/apache/datasketches/theta/EmptyCompactSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/EmptyCompactSketch.java
@@ -80,7 +80,7 @@ final class EmptyCompactSketch extends CompactSketch {
}
@Override
- public int getCurrentBytes(final boolean compact) {
+ public int getCurrentBytes() {
return 8;
}
diff --git a/src/main/java/org/apache/datasketches/theta/HeapAlphaSketch.java b/src/main/java/org/apache/datasketches/theta/HeapAlphaSketch.java
index d41beb4..5b1bb0c 100644
--- a/src/main/java/org/apache/datasketches/theta/HeapAlphaSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/HeapAlphaSketch.java
@@ -269,7 +269,7 @@ final class HeapAlphaSketch extends HeapUpdateSketch {
@Override
int getCurrentPreambleLongs(final boolean compact) {
if (!compact) { return Family.ALPHA.getMinPreLongs(); }
- return CompactOperations.computeCompactPreLongs(thetaLong_, empty_, curCount_);
+ return CompactOperations.computeCompactPreLongs(empty_, curCount_, thetaLong_);
}
@Override
diff --git a/src/main/java/org/apache/datasketches/theta/HeapCompactSketch.java b/src/main/java/org/apache/datasketches/theta/HeapCompactSketch.java
index 00f8939..82decd5 100644
--- a/src/main/java/org/apache/datasketches/theta/HeapCompactSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/HeapCompactSketch.java
@@ -68,7 +68,7 @@ class HeapCompactSketch extends CompactSketch {
cache_ = cache;
//computed
thetaLong_ = correctThetaOnCompact(empty, curCount, thetaLong);
- preLongs_ = computeCompactPreLongs(thetaLong, empty, curCount); //considers singleItem
+ preLongs_ = computeCompactPreLongs(empty, curCount, thetaLong); //considers singleItem
singleItem_ = isSingleItem(empty, curCount, thetaLong);
checkIllegalCurCountAndEmpty(empty, curCount);
}
@@ -87,7 +87,7 @@ class HeapCompactSketch extends CompactSketch {
}
@Override
- public int getCurrentBytes(final boolean compact) { //already compact; ignored
+ public int getCurrentBytes() {
return (preLongs_ + curCount_) << 3;
}
@@ -156,7 +156,7 @@ class HeapCompactSketch extends CompactSketch {
//use of Memory is convenient. The byteArray and Memory are loaded simulaneously.
@Override
public byte[] toByteArray() {
- final int bytes = getCurrentBytes(true);
+ final int bytes = getCurrentBytes();
final byte[] byteArray = new byte[bytes];
final WritableMemory dstMem = WritableMemory.wrap(byteArray);
final int emptyBit = isEmpty() ? EMPTY_FLAG_MASK : 0;
diff --git a/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java b/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java
index 1090d9e..e563c88 100644
--- a/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java
@@ -215,7 +215,7 @@ class HeapQuickSelectSketch extends HeapUpdateSketch {
@Override
int getCurrentPreambleLongs(final boolean compact) {
if (!compact) { return preambleLongs_; }
- return CompactOperations.computeCompactPreLongs(thetaLong_, empty_, curCount_);
+ return CompactOperations.computeCompactPreLongs(empty_, curCount_, thetaLong_);
}
//only used by ConcurrentHeapThetaBuffer & Test
diff --git a/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java b/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java
index 5c642ec..c214042 100644
--- a/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java
@@ -20,6 +20,8 @@
package org.apache.datasketches.theta;
import static org.apache.datasketches.Util.MIN_LG_NOM_LONGS;
+import static org.apache.datasketches.theta.CompactOperations.checkIllegalCurCountAndEmpty;
+import static org.apache.datasketches.theta.CompactOperations.correctThetaOnCompact;
import static org.apache.datasketches.theta.PreambleUtil.EMPTY_FLAG_MASK;
import static org.apache.datasketches.theta.PreambleUtil.SER_VER;
import static org.apache.datasketches.theta.PreambleUtil.insertCurCount;
@@ -59,9 +61,16 @@ abstract class HeapUpdateSketch extends UpdateSketch {
//Sketch
@Override
- public int getCurrentBytes(final boolean compact) {
- final int preLongs = getCurrentPreambleLongs(compact);
- final int dataLongs = getCurrentDataLongs(compact);
+ public int getCompactBytes() {
+ final int preLongs = getCurrentPreambleLongs(true);
+ final int dataLongs = getRetainedEntries(true);
+ return (preLongs + dataLongs) << 3;
+ }
+
+ @Override
+ public int getCurrentBytes() {
+ final int preLongs = getCurrentPreambleLongs(false);
+ final int dataLongs = getCurrentDataLongs();
return (preLongs + dataLongs) << 3;
}
@@ -106,9 +115,9 @@ abstract class HeapUpdateSketch extends UpdateSketch {
byte[] toByteArray(final int preLongs, final byte familyID) {
if (isDirty()) { rebuild(); }
- CompactOperations.checkIllegalCurCountAndEmpty(isEmpty(), getRetainedEntries(true));
+ checkIllegalCurCountAndEmpty(isEmpty(), getRetainedEntries(true));
final int preBytes = (preLongs << 3) & 0X3F;
- final int dataBytes = getCurrentDataLongs(false) << 3;
+ final int dataBytes = getCurrentDataLongs() << 3;
final byte[] byteArrOut = new byte[preBytes + dataBytes];
final WritableMemory memOut = WritableMemory.wrap(byteArrOut);
@@ -125,7 +134,7 @@ abstract class HeapUpdateSketch extends UpdateSketch {
insertCurCount(memOut, this.getRetainedEntries(true));
insertP(memOut, getP());
final long thetaLong =
- CompactOperations.correctThetaOnCompact(isEmpty(), getRetainedEntries(), getThetaLong());
+ correctThetaOnCompact(isEmpty(), getRetainedEntries(), getThetaLong());
insertThetaLong(memOut, thetaLong);
//Flags: BigEnd=0, ReadOnly=0, Empty=X, compact=0, ordered=0
diff --git a/src/main/java/org/apache/datasketches/theta/SingleItemSketch.java b/src/main/java/org/apache/datasketches/theta/SingleItemSketch.java
index 512c81c..94a9bd2 100644
--- a/src/main/java/org/apache/datasketches/theta/SingleItemSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/SingleItemSketch.java
@@ -325,7 +325,7 @@ final class SingleItemSketch extends CompactSketch {
}
@Override
- public int getCurrentBytes(final boolean compact) {
+ public int getCurrentBytes() {
return 16;
}
diff --git a/src/main/java/org/apache/datasketches/theta/Sketch.java b/src/main/java/org/apache/datasketches/theta/Sketch.java
index e87b515..eac37d0 100644
--- a/src/main/java/org/apache/datasketches/theta/Sketch.java
+++ b/src/main/java/org/apache/datasketches/theta/Sketch.java
@@ -220,6 +220,14 @@ public abstract class Sketch {
public abstract CompactSketch compact(final boolean dstOrdered, final WritableMemory dstMem);
/**
+ * Returns the number of storage bytes required for this Sketch if its current state were
+ * compacted. It this sketch is already in the compact form this is equivalent to
+ * calling {@link #getCurrentBytes()}.
+ * @return number of compact bytes
+ */
+ public abstract int getCompactBytes();
+
+ /**
* Gets the number of hash values less than the given theta.
* @param theta the given theta as a double between zero and one.
* @return the number of hash values less than the given theta.
@@ -242,11 +250,10 @@ public abstract class Sketch {
/**
* Returns the number of storage bytes required for this Sketch in its current state.
- * @param compact if true, returns the bytes required for compact form.
- * If this sketch is already in compact form this parameter is ignored.
+ *
* @return the number of storage bytes required for this sketch
*/
- public abstract int getCurrentBytes(boolean compact);
+ public abstract int getCurrentBytes();
/**
* Gets the unique count estimate.
@@ -554,6 +561,8 @@ public abstract class Sketch {
*/
abstract long[] getCache();
+ abstract int getCurrentDataLongs();
+
int getCurrentDataLongs(final boolean compact) {
return (isCompact() || compact)
? getRetainedEntries(true)
diff --git a/src/main/java/org/apache/datasketches/theta/UpdateSketch.java b/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
index f94d8b7..de519bf 100644
--- a/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
@@ -139,8 +139,18 @@ public abstract class UpdateSketch extends Sketch {
public CompactSketch compact(final boolean dstOrdered, final WritableMemory dstMem) {
return componentsToCompact(getThetaLong(), getRetainedEntries(), getSeedHash(), isEmpty(),
false, false, dstOrdered, dstMem, getCache());
+ }
- //return compact(this, dstOrdered, dstMem);
+ @Override
+ public int getCompactBytes() {
+ final int preLongs = getCurrentPreambleLongs(true);
+ final int dataLongs = getRetainedEntries();
+ return (preLongs + dataLongs) << 3;
+ }
+
+ @Override
+ int getCurrentDataLongs() {
+ return 1 << getLgArrLongs();
}
@Override
diff --git a/src/test/java/org/apache/datasketches/theta/AnotBimplTest.java b/src/test/java/org/apache/datasketches/theta/AnotBimplTest.java
index 4341be0..3ad0559 100644
--- a/src/test/java/org/apache/datasketches/theta/AnotBimplTest.java
+++ b/src/test/java/org/apache/datasketches/theta/AnotBimplTest.java
@@ -70,7 +70,7 @@ public class AnotBimplTest {
assertEquals(rsk1.getEstimate(), k/2.0);
//getCurrentBytes( compact )
- int bytes = rsk1.getCurrentBytes(true);
+ int bytes = rsk1.getCurrentBytes();
byte[] byteArray = new byte[bytes];
WritableMemory mem = WritableMemory.wrap(byteArray);
@@ -329,7 +329,7 @@ public class AnotBimplTest {
c.update(i); //256
}
- int memBytes = Sketch.getMaxCompactSketchBytes(a.getCurrentBytes(true));
+ int memBytes = Sketch.getMaxCompactSketchBytes(a.getRetainedEntries(true));
byte[] memArr = new byte[memBytes];
WritableMemory mem = WritableMemory.wrap(memArr);
@@ -356,7 +356,7 @@ public class AnotBimplTest {
UpdateSketch skB = Sketches.updateSketchBuilder().build();
AnotB aNotB = Sketches.setOperationBuilder().buildANotB();
CompactSketch csk = aNotB.aNotB(skA, skB);
- assertEquals(csk.getCurrentBytes(true), 8);
+ assertEquals(csk.getCurrentBytes(), 8);
}
@Test
@@ -367,7 +367,7 @@ public class AnotBimplTest {
AnotB aNotB = Sketches.setOperationBuilder().buildANotB();
aNotB.update(skA, skB);
CompactSketch csk = aNotB.getResult();
- assertEquals(csk.getCurrentBytes(true), 8);
+ assertEquals(csk.getCurrentBytes(), 8);
}
@Test
diff --git a/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java b/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
index a129766..a2807cb 100644
--- a/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
@@ -76,7 +76,7 @@ public class CompactSketchTest {
/****OFF HEAP MEMORY -- WRAP****/
//Prepare Memory for direct
- int bytes = usk.getCurrentBytes(true); //for Compact
+ int bytes = usk.getCompactBytes(); //for Compact
try (WritableDirectHandle wdh = WritableMemory.allocateDirect(bytes)) {
WritableMemory directMem = wdh.get();
@@ -114,7 +114,7 @@ public class CompactSketchTest {
assertEquals(testSk.getSeedHash(), 0);
assertEquals(testSk.getRetainedEntries(true), 0);
assertEquals(testSk.getEstimate(), 0.0, 0.0);
- assertEquals(testSk.getCurrentBytes(true), 8);
+ assertEquals(testSk.getCurrentBytes(), 8);
assertNotNull(testSk.iterator());
assertEquals(testSk.toByteArray().length, 8);
assertEquals(testSk.getCache().length, 0);
@@ -132,7 +132,7 @@ public class CompactSketchTest {
assertEquals(testSk.getSeedHash(), refSk.getSeedHash());
assertEquals(testSk.getRetainedEntries(true), 1);
assertEquals(testSk.getEstimate(), 1.0, 0.0);
- assertEquals(testSk.getCurrentBytes(true), 16);
+ assertEquals(testSk.getCurrentBytes(), 16);
assertNotNull(testSk.iterator());
assertEquals(testSk.toByteArray().length, 16);
assertEquals(testSk.getCache().length, 1);
@@ -164,7 +164,7 @@ public class CompactSketchTest {
assertEquals(testSk.getSeedHash(), refSk.getSeedHash());
assertEquals(testSk.getRetainedEntries(true), refSk.getRetainedEntries());
assertEquals(testSk.getEstimate(), refSk.getEstimate(), 0.0);
- assertEquals(testSk.getCurrentBytes(true), refSk.getCurrentBytes(true));
+ assertEquals(testSk.getCurrentBytes(), refSk.getCurrentBytes());
assertEquals(testSk.toByteArray().length, refSk.toByteArray().length);
assertEquals(testSk.getCache().length, refSk.getCache().length);
assertEquals(testSk.getCurrentPreambleLongs(true), refSk.getCurrentPreambleLongs(true));
@@ -174,7 +174,7 @@ public class CompactSketchTest {
public void checkDirectSingleItemSketch() {
UpdateSketch sk = Sketches.updateSketchBuilder().build();
sk.update(1);
- int bytes = sk.getCurrentBytes(true);
+ int bytes = sk.getCompactBytes();
WritableMemory wmem = WritableMemory.allocate(bytes);
sk.compact(true, wmem);
Sketch csk2 = Sketch.heapify(wmem);
@@ -192,7 +192,7 @@ public class CompactSketchTest {
usk.update(i);
}
- int bytes = usk.getCurrentBytes(compact);
+ int bytes = usk.getCompactBytes();
byte[] byteArray = new byte[bytes -8]; //too small
WritableMemory mem = WritableMemory.wrap(byteArray);
usk.compact(ordered, mem);
@@ -209,7 +209,7 @@ public class CompactSketchTest {
usk.update(i);
}
- int bytes = usk.getCurrentBytes(compact);
+ int bytes = usk.getCompactBytes();
byte[] byteArray = new byte[bytes -8]; //too small
WritableMemory mem = WritableMemory.wrap(byteArray);
usk.compact(ordered, mem);
@@ -319,7 +319,7 @@ public class CompactSketchTest {
public void checkGetCache() {
UpdateSketch sk = Sketches.updateSketchBuilder().setP((float).5).build();
sk.update(7);
- int bytes = sk.getCurrentBytes(true);
+ int bytes = sk.getCompactBytes();
CompactSketch csk = sk.compact(true, WritableMemory.allocate(bytes));
long[] cache = csk.getCache();
assertTrue(cache.length == 0);
@@ -353,7 +353,7 @@ public class CompactSketchTest {
void check(CompactSketch csk) {
assertEquals(csk.getClass().getSimpleName(), classType, "ClassType");
assertEquals(csk.getRetainedEntries(), count, "curCount");
- assertEquals(csk.getCurrentBytes(true), bytes, "Bytes" );
+ assertEquals(csk.getCurrentBytes(), bytes, "Bytes" );
assertEquals(csk.isCompact(), compact, "Compact");
assertEquals(csk.isEmpty(), empty, "Empty");
assertEquals(csk.isDirect(), direct, "Direct");
diff --git a/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java
index acca0b6..5bb93e8 100644
--- a/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java
@@ -382,7 +382,7 @@ public class ConcurrentDirectQuickSelectSketchTest {
assertEquals(csk.isEstimationMode(), estimating);
assertEquals(csk.getClass().getSimpleName(), "HeapCompactSketch");
- int bytes = local.getCurrentBytes(true);
+ int bytes = local.getCurrentBytes(); //TODO WHAT IS GOING ON HERE
assertEquals(bytes, (k*8) + (Family.COMPACT.getMaxPreLongs() << 3));
byte[] memArr2 = new byte[bytes];
WritableMemory mem2 = WritableMemory.wrap(memArr2);
@@ -427,7 +427,7 @@ public class ConcurrentDirectQuickSelectSketchTest {
double localUB = local.getUpperBound(2);
assertFalse(local.isEstimationMode());
- int bytes = local.getCurrentBytes(true); //compact form
+ int bytes = local.getCompactBytes(); //compact form
assertEquals(bytes, 8);
byte[] memArr2 = new byte[bytes];
WritableMemory mem2 = WritableMemory.wrap(memArr2);
diff --git a/src/test/java/org/apache/datasketches/theta/SketchTest.java b/src/test/java/org/apache/datasketches/theta/SketchTest.java
index db8ac34..840b31e 100644
--- a/src/test/java/org/apache/datasketches/theta/SketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/SketchTest.java
@@ -30,6 +30,7 @@ import static org.apache.datasketches.Util.DEFAULT_NOMINAL_ENTRIES;
import static org.apache.datasketches.Util.DEFAULT_UPDATE_SEED;
import static org.apache.datasketches.theta.BackwardConversions.convertSerVer3toSerVer1;
import static org.apache.datasketches.theta.BackwardConversions.convertSerVer3toSerVer2;
+import static org.apache.datasketches.theta.CompactOperations.computeCompactPreLongs;
import static org.apache.datasketches.theta.PreambleUtil.COMPACT_FLAG_MASK;
import static org.apache.datasketches.theta.PreambleUtil.FLAGS_BYTE;
import static org.apache.datasketches.theta.PreambleUtil.READ_ONLY_FLAG_MASK;
@@ -66,17 +67,17 @@ public class SketchTest {
UpdateSketch sketch = UpdateSketch.builder().setNominalEntries(k).build(); // QS Sketch
assertEquals(sketch.getCurrentPreambleLongs(false), lowQSPreLongs);
assertEquals(sketch.getCurrentPreambleLongs(true), 1); //compact form
- assertEquals(sketch.getCurrentDataLongs(false), k*2);
- assertEquals(sketch.getCurrentDataLongs(true), 0); //compact form
+ assertEquals(sketch.getCurrentDataLongs(), k*2);
assertEquals(sketch.getCurrentBytes(false), (k*2*8) + (lowQSPreLongs << 3));
assertEquals(sketch.getCurrentBytes(true), lowCompPreLongs << 3);
CompactSketch compSk = sketch.compact(false, null);
assertEquals(compSk.getCurrentBytes(true), 8);
assertEquals(compSk.getCurrentBytes(false), 8);
+ assertEquals(compSk.getCurrentDataLongs(), 0);
- int compPreLongs = CompactOperations.computeCompactPreLongs(sketch.getThetaLong(), sketch.isEmpty(),
- sketch.getRetainedEntries(true));
+ int compPreLongs = computeCompactPreLongs(sketch.isEmpty(), sketch.getRetainedEntries(true),
+ sketch.getThetaLong());
assertEquals(compPreLongs, 1);
for (int i=0; i<k; i++) {
@@ -85,13 +86,12 @@ public class SketchTest {
assertEquals(sketch.getCurrentPreambleLongs(false), lowQSPreLongs);
assertEquals(sketch.getCurrentPreambleLongs(true), 2); //compact form
- assertEquals(sketch.getCurrentDataLongs(false), k*2);
- assertEquals(sketch.getCurrentDataLongs(true), k); //compact form
+ assertEquals(sketch.getCurrentDataLongs(), k*2);
assertEquals(sketch.getCurrentBytes(false), (k*2*8) + (lowQSPreLongs << 3));
assertEquals(sketch.getCurrentBytes(true), (k*8) + (2*8)); //compact form //FAILS HERE
- compPreLongs = CompactOperations.computeCompactPreLongs(sketch.getThetaLong(), sketch.isEmpty(),
- sketch.getRetainedEntries(true));
+ compPreLongs = computeCompactPreLongs(sketch.isEmpty(), sketch.getRetainedEntries(true),
+ sketch.getThetaLong());
assertEquals(compPreLongs, 2);
for (int i = k; i < (2*k); i++) {
@@ -101,13 +101,12 @@ public class SketchTest {
assertEquals(sketch.getCurrentPreambleLongs(false), lowQSPreLongs);
assertEquals(sketch.getCurrentPreambleLongs(true), 3); //compact form
- assertEquals(sketch.getCurrentDataLongs(false), k*2);
- assertEquals(sketch.getCurrentDataLongs(true), curCount); //compact form
+ assertEquals(sketch.getCurrentDataLongs(), k*2);
assertEquals(sketch.getCurrentBytes(false), (k*2*8) + (lowQSPreLongs << 3));
assertEquals(sketch.getCurrentBytes(true), (curCount*8) + (3*8)); //compact form
- compPreLongs = CompactOperations.computeCompactPreLongs(sketch.getThetaLong(), sketch.isEmpty(),
- sketch.getRetainedEntries(true));
+ compPreLongs = computeCompactPreLongs(sketch.isEmpty(), sketch.getRetainedEntries(true),
+ sketch.getThetaLong());
assertEquals(compPreLongs, 3);
for (int i=0; i<3; i++) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[incubator-datasketches-java] 06/10: Interm: Completed major
rewrite of Concurrent Theta Sketch unit tests and cleaned up main code as
well.
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leerho pushed a commit to branch Refactor_Theta_Tuple
in repository https://gitbox.apache.org/repos/asf/incubator-datasketches-java.git
commit 8736b03d8b2ae49c3f2b1e26fa6c89127d39c2c4
Author: Lee Rhodes <le...@users.noreply.github.com>
AuthorDate: Wed Jul 1 23:00:22 2020 -0700
Interm: Completed major rewrite of Concurrent Theta Sketch unit tests
and cleaned up main code as well.
---
.../org/apache/datasketches/theta/AnotBimpl.java | 6 +-
.../theta/ConcurrentHeapThetaBuffer.java | 95 +-
.../theta/ConcurrentSharedThetaSketch.java | 15 +-
.../datasketches/theta/DirectCompactSketch.java | 9 +-
.../theta/DirectQuickSelectSketch.java | 2 +-
.../theta/DirectQuickSelectSketchR.java | 8 +-
.../datasketches/theta/EmptyCompactSketch.java | 7 +-
.../apache/datasketches/theta/HeapAlphaSketch.java | 8 +-
.../datasketches/theta/HeapCompactSketch.java | 11 +-
.../datasketches/theta/HeapQuickSelectSketch.java | 8 +-
.../datasketches/theta/HeapUpdateSketch.java | 12 +-
.../datasketches/theta/JaccardSimilarity.java | 12 +-
.../datasketches/theta/SingleItemSketch.java | 7 +-
.../java/org/apache/datasketches/theta/Sketch.java | 17 +-
.../apache/datasketches/theta/UpdateSketch.java | 6 +-
.../datasketches/theta/UpdateSketchBuilder.java | 6 +-
.../java/org/apache/datasketches/tuple/AnotB.java | 4 +-
.../apache/datasketches/tuple/Intersection.java | 6 +-
.../datasketches/theta/BackwardConversions.java | 4 +-
.../datasketches/theta/CompactSketchTest.java | 10 +-
.../ConcurrentDirectQuickSelectSketchTest.java | 1142 +++++++++-----------
.../theta/ConcurrentHeapQuickSelectSketchTest.java | 482 ++++-----
.../theta/DirectQuickSelectSketchTest.java | 2 +-
.../apache/datasketches/theta/DirectUnionTest.java | 4 +-
.../datasketches/theta/SetOpsCornerCasesTest.java | 28 +-
.../datasketches/theta/SingleItemSketchTest.java | 2 +-
.../org/apache/datasketches/theta/SketchTest.java | 12 +-
.../apache/datasketches/theta/UnionImplTest.java | 2 +-
28 files changed, 923 insertions(+), 1004 deletions(-)
diff --git a/src/main/java/org/apache/datasketches/theta/AnotBimpl.java b/src/main/java/org/apache/datasketches/theta/AnotBimpl.java
index 752baac..bccb455 100644
--- a/src/main/java/org/apache/datasketches/theta/AnotBimpl.java
+++ b/src/main/java/org/apache/datasketches/theta/AnotBimpl.java
@@ -92,7 +92,7 @@ final class AnotBimpl extends AnotB {
thetaLong_ = Math.min(thetaLong_, thetaLongB);
//Build hashtable and removes hashes of skB >= theta
- final int countB = skB.getRetainedEntries();
+ final int countB = skB.getRetainedEntries(true);
CompactSketch cskB = null;
UpdateSketch uskB = null;
final long[] hashTableB;
@@ -161,12 +161,12 @@ final class AnotBimpl extends AnotB {
? (CompactSketch)skA
: ((UpdateSketch)skA).compact();
final long[] hashArrA = cskA.getCache().clone();
- final int countA = cskA.getRetainedEntries();
+ final int countA = cskA.getRetainedEntries(true);
//Compare with skB
final long thetaLongB = skB.getThetaLong();
final long thetaLong = Math.min(thetaLongA, thetaLongB);
- final int countB = skB.getRetainedEntries();
+ final int countB = skB.getRetainedEntries(true);
//Rebuild hashtable and removes hashes of skB >= thetaLong
final long[] hashTableB = convertToHashTable(skB.getCache(), countB, thetaLong, REBUILD_THRESHOLD);
diff --git a/src/main/java/org/apache/datasketches/theta/ConcurrentHeapThetaBuffer.java b/src/main/java/org/apache/datasketches/theta/ConcurrentHeapThetaBuffer.java
index e4b18f2..e397ecc 100644
--- a/src/main/java/org/apache/datasketches/theta/ConcurrentHeapThetaBuffer.java
+++ b/src/main/java/org/apache/datasketches/theta/ConcurrentHeapThetaBuffer.java
@@ -29,14 +29,25 @@ import org.apache.datasketches.HashOperations;
import org.apache.datasketches.ResizeFactor;
/**
- * The theta filtering buffer that operates in the context of a single writing thread.
- * This is a bounded size filter. When the buffer becomes full its content is propagated into the
- * shared sketch.
- * The limit on the buffer size is configurable. Bound of size 1 allows to maintain error bound
- * that is close to the error bound of a sequential theta sketch.
+ * This is a theta filtering, bounded size buffer that operates in the context of a single writing
+ * thread. When the buffer becomes full its content is propagated into the shared sketch, which
+ * may be on a different thread. The limit on the buffer size is configurable. A bound of size 1
+ * allows the combination of buffers and shared sketch to maintain an error bound in real-time
+ * that is close to the error bound of a sequential theta sketch. Allowing larger buffer sizes
+ * enables amortization of the cost propagations and substantially improves overall system throughput.
+ * The error caused by the buffering is essentially a perspecitive of time and synchronization
+ * and not really a true error. At the end of a stream, after all the buffers have synchronized with
+ * the shared sketch, there is no additional error.
* Propagation is done either synchronously by the updating thread, or asynchronously by a
* background propagation thread.
*
+ * <p>This is a buffer, not a sketch, and it extends the <i>HeapQuickSelectSketch</i>
+ * in order to leverage some of the sketch machinery to make its work simple. However, if this
+ * buffer receives a query, like <i>getEstimate()</i>, the correct answer does not come from the super
+ * <i>HeapQuickSelectSketch</i>, which knows nothing about the concurrency relationship to the
+ * shared concurrent sketch, it must come from the shared concurrent sketch. As a result nearly all
+ * of the inherited sketch methods are redirected to the shared concurrent sketch.
+ *
* @author eshcar
* @author Lee Rhodes
*/
@@ -75,7 +86,41 @@ final class ConcurrentHeapThetaBuffer extends HeapQuickSelectSketch {
return Math.min(lgNomLongs, (int)Math.log(Math.sqrt(exactSize) / (2 * maxNumLocalBuffers)));
}
- //Sketch overrides
+ //concurrent restricted methods
+
+ /**
+ * Propagates a single hash value to the shared sketch
+ *
+ * @param hash to be propagated
+ */
+ private boolean propagateToSharedSketch(final long hash) {
+ //noinspection StatementWithEmptyBody
+ while (localPropagationInProgress.get()) {
+ } //busy wait until previous propagation completed
+ localPropagationInProgress.set(true);
+ final boolean res = shared.propagate(localPropagationInProgress, null, hash);
+ //in this case the parent empty_ and curCount_ were not touched
+ thetaLong_ = shared.getVolatileTheta();
+ return res;
+ }
+
+ /**
+ * Propagates the content of the buffer as a sketch to the shared sketch
+ */
+ private void propagateToSharedSketch() {
+ //noinspection StatementWithEmptyBody
+ while (localPropagationInProgress.get()) {
+ } //busy wait until previous propagation completed
+
+ final CompactSketch compactSketch = compact(propagateOrderedCompact, null);
+ localPropagationInProgress.set(true);
+ shared.propagate(localPropagationInProgress, compactSketch,
+ ConcurrentSharedThetaSketch.NOT_SINGLE_HASH);
+ super.reset();
+ thetaLong_ = shared.getVolatileTheta();
+ }
+
+ //Public Sketch overrides proxies to shared concurrent sketch
@Override
public int getCompactBytes() {
@@ -122,12 +167,14 @@ final class ConcurrentHeapThetaBuffer extends HeapQuickSelectSketch {
return shared.isEstimationMode();
}
+ //End of proxies
+
@Override
public byte[] toByteArray() {
throw new UnsupportedOperationException("Local theta buffer need not be serialized");
}
- //UpdateSketch overrides
+ //Public UpdateSketch overrides
@Override
public void reset() {
@@ -136,7 +183,7 @@ final class ConcurrentHeapThetaBuffer extends HeapQuickSelectSketch {
localPropagationInProgress.set(false);
}
- //restricted methods
+ //Restricted UpdateSketch overrides
/**
* Updates buffer with given hash value.
@@ -163,7 +210,7 @@ final class ConcurrentHeapThetaBuffer extends HeapQuickSelectSketch {
}
}
final UpdateReturnState state = super.hashUpdate(hash);
- if (isOutOfSpace(getRetainedEntries() + 1)) {
+ if (isOutOfSpace(getRetainedEntries(true) + 1)) {
propagateToSharedSketch();
return ConcurrentPropagated;
}
@@ -173,35 +220,5 @@ final class ConcurrentHeapThetaBuffer extends HeapQuickSelectSketch {
return state;
}
- /**
- * Propagates a single hash value to the shared sketch
- *
- * @param hash to be propagated
- */
- private boolean propagateToSharedSketch(final long hash) {
- //noinspection StatementWithEmptyBody
- while (localPropagationInProgress.get()) {
- } //busy wait until previous propagation completed
- localPropagationInProgress.set(true);
- final boolean res = shared.propagate(localPropagationInProgress, null, hash);
- //in this case the parent empty_ and curCount_ were not touched
- thetaLong_ = shared.getVolatileTheta();
- return res;
- }
-
- /**
- * Propagates the content of the buffer as a sketch to the shared sketch
- */
- private void propagateToSharedSketch() {
- //noinspection StatementWithEmptyBody
- while (localPropagationInProgress.get()) {
- } //busy wait until previous propagation completed
- final CompactSketch compactSketch = compact(propagateOrderedCompact, null);
- localPropagationInProgress.set(true);
- shared.propagate(localPropagationInProgress, compactSketch,
- ConcurrentSharedThetaSketch.NOT_SINGLE_HASH);
- super.reset();
- thetaLong_ = shared.getVolatileTheta();
- }
}
diff --git a/src/main/java/org/apache/datasketches/theta/ConcurrentSharedThetaSketch.java b/src/main/java/org/apache/datasketches/theta/ConcurrentSharedThetaSketch.java
index 3c2c4b1..49a8140 100644
--- a/src/main/java/org/apache/datasketches/theta/ConcurrentSharedThetaSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/ConcurrentSharedThetaSketch.java
@@ -24,9 +24,9 @@ import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.datasketches.memory.WritableMemory;
/**
- * An interface to define the API of a concurrent shared theta sketch.
+ * An internal interface to define the API of a concurrent shared theta sketch.
* It reflects all data processed by a single or multiple update threads, and can serve queries at
- * any time
+ * any time.
*
* @author eshcar
*/
@@ -117,6 +117,17 @@ interface ConcurrentSharedThetaSketch {
//The following mirrors are public methods that already exist on the "extends" side of the dual
// inheritance. They are provided here to allow casts to this interface access
// to these methods without having to cast back to the extended parent class.
+ //
+ //This allows an internal class to cast either the Concurrent Direct or Concurrent Heap
+ //shared class to this interface and have access to the above special concurrent methods as
+ //well as the methods below.
+ //
+ //For the external user all of the below methods can be obtained by casting the shared
+ //sketch to UpdateSketch. However, these methods here also act as an alias so that an
+ //attempt to access these methods from the local buffer will be divered to the shared
+ //sketch.
+
+ //From Sketch
int getCompactBytes();
diff --git a/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java b/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java
index 458a93d..fc78cb0 100644
--- a/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/DirectCompactSketch.java
@@ -142,7 +142,7 @@ class DirectCompactSketch extends CompactSketch {
@Override
public HashIterator iterator() {
- return new MemoryHashIterator(mem_, getRetainedEntries(), getThetaLong());
+ return new MemoryHashIterator(mem_, getRetainedEntries(true), getThetaLong());
}
@Override
@@ -172,7 +172,12 @@ class DirectCompactSketch extends CompactSketch {
}
@Override
- int getCurrentPreambleLongs(final boolean compact) { //already compact; ignore
+ int getCompactPreambleLongs() {
+ return extractPreLongs(mem_);
+ }
+
+ @Override
+ int getCurrentPreambleLongs() {
return extractPreLongs(mem_);
}
diff --git a/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketch.java b/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketch.java
index f80473a..9309194 100644
--- a/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketch.java
@@ -263,7 +263,7 @@ class DirectQuickSelectSketch extends DirectQuickSelectSketchR {
return RejectedDuplicate; //Duplicate, not inserted
}
//insertion occurred, increment curCount
- final int curCount = getRetainedEntries() + 1;
+ final int curCount = getRetainedEntries(true) + 1;
wmem_.putInt(RETAINED_ENTRIES_INT, curCount); //update curCount
if (isOutOfSpace(curCount)) { //we need to do something, we are out of space
diff --git a/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java b/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java
index 36f4da3..d5290b1 100644
--- a/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java
+++ b/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java
@@ -215,12 +215,16 @@ class DirectQuickSelectSketchR extends UpdateSketch {
}
@Override
- int getCurrentPreambleLongs(final boolean compact) {
- if (!compact) { return PreambleUtil.extractPreLongs(wmem_); }
+ int getCompactPreambleLongs() {
return computeCompactPreLongs(isEmpty(), getRetainedEntries(true), getThetaLong());
}
@Override
+ int getCurrentPreambleLongs() {
+ return PreambleUtil.extractPreLongs(wmem_);
+ }
+
+ @Override
WritableMemory getMemory() {
return wmem_;
}
diff --git a/src/main/java/org/apache/datasketches/theta/EmptyCompactSketch.java b/src/main/java/org/apache/datasketches/theta/EmptyCompactSketch.java
index 54c71eb..ffdb668 100644
--- a/src/main/java/org/apache/datasketches/theta/EmptyCompactSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/EmptyCompactSketch.java
@@ -138,7 +138,12 @@ final class EmptyCompactSketch extends CompactSketch {
}
@Override
- int getCurrentPreambleLongs(final boolean compact) {
+ int getCompactPreambleLongs() {
+ return 1;
+ }
+
+ @Override
+ int getCurrentPreambleLongs() {
return 1;
}
diff --git a/src/main/java/org/apache/datasketches/theta/HeapAlphaSketch.java b/src/main/java/org/apache/datasketches/theta/HeapAlphaSketch.java
index 5b1bb0c..2561b44 100644
--- a/src/main/java/org/apache/datasketches/theta/HeapAlphaSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/HeapAlphaSketch.java
@@ -267,12 +267,16 @@ final class HeapAlphaSketch extends HeapUpdateSketch {
//restricted methods
@Override
- int getCurrentPreambleLongs(final boolean compact) {
- if (!compact) { return Family.ALPHA.getMinPreLongs(); }
+ int getCompactPreambleLongs() {
return CompactOperations.computeCompactPreLongs(empty_, curCount_, thetaLong_);
}
@Override
+ int getCurrentPreambleLongs() {
+ return Family.ALPHA.getMinPreLongs();
+ }
+
+ @Override
WritableMemory getMemory() {
return null;
}
diff --git a/src/main/java/org/apache/datasketches/theta/HeapCompactSketch.java b/src/main/java/org/apache/datasketches/theta/HeapCompactSketch.java
index 82decd5..b898a5e 100644
--- a/src/main/java/org/apache/datasketches/theta/HeapCompactSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/HeapCompactSketch.java
@@ -82,7 +82,7 @@ class HeapCompactSketch extends CompactSketch {
@Override
public CompactSketch compact(final boolean dstOrdered, final WritableMemory dstMem) {
- return componentsToCompact(getThetaLong(), getRetainedEntries(), getSeedHash(), isEmpty(),
+ return componentsToCompact(getThetaLong(), getRetainedEntries(true), getSeedHash(), isEmpty(),
true, ordered_, dstOrdered, dstMem, getCache());
}
@@ -139,7 +139,12 @@ class HeapCompactSketch extends CompactSketch {
}
@Override
- int getCurrentPreambleLongs(final boolean compact) { //already compact; ignored
+ int getCompactPreambleLongs() {
+ return preLongs_;
+ }
+
+ @Override
+ int getCurrentPreambleLongs() { //already compact; ignored
return preLongs_;
}
@@ -164,7 +169,7 @@ class HeapCompactSketch extends CompactSketch {
final int singleItemBit = singleItem_ ? SINGLEITEM_FLAG_MASK : 0;
final byte flags = (byte) (emptyBit | READ_ONLY_FLAG_MASK | COMPACT_FLAG_MASK
| orderedBit | singleItemBit);
- final int preLongs = getCurrentPreambleLongs(true);
+ final int preLongs = getCompactPreambleLongs();
loadCompactMemory(getCache(), getSeedHash(), getRetainedEntries(true), getThetaLong(),
dstMem, flags, preLongs);
return byteArray;
diff --git a/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java b/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java
index e563c88..d6303bf 100644
--- a/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java
@@ -213,11 +213,15 @@ class HeapQuickSelectSketch extends HeapUpdateSketch {
}
@Override
- int getCurrentPreambleLongs(final boolean compact) {
- if (!compact) { return preambleLongs_; }
+ int getCompactPreambleLongs() {
return CompactOperations.computeCompactPreLongs(empty_, curCount_, thetaLong_);
}
+ @Override
+ int getCurrentPreambleLongs() {
+ return preambleLongs_;
+ }
+
//only used by ConcurrentHeapThetaBuffer & Test
int getHashTableThreshold() {
return hashTableThreshold_;
diff --git a/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java b/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java
index c214042..d62e88d 100644
--- a/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java
@@ -61,15 +61,8 @@ abstract class HeapUpdateSketch extends UpdateSketch {
//Sketch
@Override
- public int getCompactBytes() {
- final int preLongs = getCurrentPreambleLongs(true);
- final int dataLongs = getRetainedEntries(true);
- return (preLongs + dataLongs) << 3;
- }
-
- @Override
public int getCurrentBytes() {
- final int preLongs = getCurrentPreambleLongs(false);
+ final int preLongs = getCurrentPreambleLongs();
final int dataLongs = getCurrentDataLongs();
return (preLongs + dataLongs) << 3;
}
@@ -113,6 +106,7 @@ abstract class HeapUpdateSketch extends UpdateSketch {
return Util.computeSeedHash(getSeed());
}
+ //Used by HeapAlphaSketch and HeapQuickSelectSketch
byte[] toByteArray(final int preLongs, final byte familyID) {
if (isDirty()) { rebuild(); }
checkIllegalCurCountAndEmpty(isEmpty(), getRetainedEntries(true));
@@ -134,7 +128,7 @@ abstract class HeapUpdateSketch extends UpdateSketch {
insertCurCount(memOut, this.getRetainedEntries(true));
insertP(memOut, getP());
final long thetaLong =
- correctThetaOnCompact(isEmpty(), getRetainedEntries(), getThetaLong());
+ correctThetaOnCompact(isEmpty(), getRetainedEntries(true), getThetaLong());
insertThetaLong(memOut, thetaLong);
//Flags: BigEnd=0, ReadOnly=0, Empty=X, compact=0, ordered=0
diff --git a/src/main/java/org/apache/datasketches/theta/JaccardSimilarity.java b/src/main/java/org/apache/datasketches/theta/JaccardSimilarity.java
index ce93ab3..05c62cb 100644
--- a/src/main/java/org/apache/datasketches/theta/JaccardSimilarity.java
+++ b/src/main/java/org/apache/datasketches/theta/JaccardSimilarity.java
@@ -59,8 +59,8 @@ public final class JaccardSimilarity {
if (sketchA.isEmpty() && sketchB.isEmpty()) { return ONES.clone(); }
if (sketchA.isEmpty() || sketchB.isEmpty()) { return ZEROS.clone(); }
- final int countA = sketchA.getRetainedEntries();
- final int countB = sketchB.getRetainedEntries();
+ final int countA = sketchA.getRetainedEntries(true);
+ final int countB = sketchB.getRetainedEntries(true);
//Create the Union
final int minK = 1 << MIN_LG_NOM_LONGS;
@@ -74,7 +74,7 @@ public final class JaccardSimilarity {
final long thetaLongUAB = unionAB.getThetaLong();
final long thetaLongA = sketchA.getThetaLong();
final long thetaLongB = sketchB.getThetaLong();
- final int countUAB = unionAB.getRetainedEntries();
+ final int countUAB = unionAB.getRetainedEntries(true);
//Check for identical data
if ((countUAB == countA) && (countUAB == countB)
@@ -110,8 +110,8 @@ public final class JaccardSimilarity {
if (sketchA.isEmpty() && sketchB.isEmpty()) { return true; }
if (sketchA.isEmpty() || sketchB.isEmpty()) { return false; }
- final int countA = sketchA.getRetainedEntries();
- final int countB = sketchB.getRetainedEntries();
+ final int countA = sketchA.getRetainedEntries(true);
+ final int countB = sketchB.getRetainedEntries(true);
//Create the Union
final Union union =
@@ -122,7 +122,7 @@ public final class JaccardSimilarity {
final long thetaLongUAB = unionAB.getThetaLong();
final long thetaLongA = sketchA.getThetaLong();
final long thetaLongB = sketchB.getThetaLong();
- final int countUAB = unionAB.getRetainedEntries();
+ final int countUAB = unionAB.getRetainedEntries(true);
//Check for identical counts and thetas
if ((countUAB == countA) && (countUAB == countB)
diff --git a/src/main/java/org/apache/datasketches/theta/SingleItemSketch.java b/src/main/java/org/apache/datasketches/theta/SingleItemSketch.java
index 94a9bd2..cacd15e 100644
--- a/src/main/java/org/apache/datasketches/theta/SingleItemSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/SingleItemSketch.java
@@ -395,7 +395,12 @@ final class SingleItemSketch extends CompactSketch {
}
@Override
- int getCurrentPreambleLongs(final boolean compact) {
+ int getCompactPreambleLongs() {
+ return 1;
+ }
+
+ @Override
+ int getCurrentPreambleLongs() {
return 1;
}
diff --git a/src/main/java/org/apache/datasketches/theta/Sketch.java b/src/main/java/org/apache/datasketches/theta/Sketch.java
index aea0657..251abc8 100644
--- a/src/main/java/org/apache/datasketches/theta/Sketch.java
+++ b/src/main/java/org/apache/datasketches/theta/Sketch.java
@@ -573,15 +573,24 @@ public abstract class Sketch {
*/
abstract long[] getCache();
+ /**
+ * Gets preamble longs if stored in compact form. If this sketch is already in compact form,
+ * this is identical to the call {@link #getCurrentPreambleLongs()}.
+ * @return preamble longs if stored in compact form.
+ */
+ abstract int getCompactPreambleLongs();
+
+ /**
+ * Gets the number of data longs if stored in current state.
+ * @return the number of data longs if stored in current state.
+ */
abstract int getCurrentDataLongs();
/**
* Returns preamble longs if stored in current state.
- * @param compact if true, returns the preamble longs required for compact form.
- * If this sketch is already in compact form this parameter is ignored.
- * @return preamble longs if stored.
+ * @return number of preamble longs if stored.
*/
- abstract int getCurrentPreambleLongs(boolean compact);
+ abstract int getCurrentPreambleLongs();
/**
* Returns the Memory object if it exists, otherwise null.
diff --git a/src/main/java/org/apache/datasketches/theta/UpdateSketch.java b/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
index bed93a5..c28c983 100644
--- a/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
@@ -137,14 +137,14 @@ public abstract class UpdateSketch extends Sketch {
@Override
public CompactSketch compact(final boolean dstOrdered, final WritableMemory dstMem) {
- return componentsToCompact(getThetaLong(), getRetainedEntries(), getSeedHash(), isEmpty(),
+ return componentsToCompact(getThetaLong(), getRetainedEntries(true), getSeedHash(), isEmpty(),
false, false, dstOrdered, dstMem, getCache());
}
@Override
public int getCompactBytes() {
- final int preLongs = getCurrentPreambleLongs(true);
- final int dataLongs = getRetainedEntries();
+ final int preLongs = getCompactPreambleLongs();
+ final int dataLongs = getRetainedEntries(true);
return (preLongs + dataLongs) << 3;
}
diff --git a/src/main/java/org/apache/datasketches/theta/UpdateSketchBuilder.java b/src/main/java/org/apache/datasketches/theta/UpdateSketchBuilder.java
index 705af29..dca3569 100644
--- a/src/main/java/org/apache/datasketches/theta/UpdateSketchBuilder.java
+++ b/src/main/java/org/apache/datasketches/theta/UpdateSketchBuilder.java
@@ -115,7 +115,7 @@ public class UpdateSketchBuilder {
* Be aware that sketches as large as this maximum value may not have been
* thoroughly tested or characterized for performance.
*
- * @param lgNomEntries the Log Nominal Entries for the concurrent shared sketch
+ * @param lgNomEntries the Log Nominal Entries. Also for the concurrent shared sketch
* @return this UpdateSketchBuilder
*/
public UpdateSketchBuilder setLogNominalEntries(final int lgNomEntries) {
@@ -139,7 +139,7 @@ public class UpdateSketchBuilder {
*
* @param nomEntries <a href="{@docRoot}/resources/dictionary.html#nomEntries">Nominal Entries</a>
* This will become the ceiling power of 2 if it is not.
- * @return this ConcurrentThetaBuilder
+ * @return this UpdateSketchBuilder
*/
public UpdateSketchBuilder setLocalNominalEntries(final int nomEntries) {
bLocalLgNomLongs = Integer.numberOfTrailingZeros(ceilingPowerOf2(nomEntries));
@@ -158,7 +158,7 @@ public class UpdateSketchBuilder {
* value have not been thoroughly tested or characterized for performance.
*
* @param lgNomEntries the Log Nominal Entries for a concurrent local sketch
- * @return this ConcurrentThetaBuilder
+ * @return this UpdateSketchBuilder
*/
public UpdateSketchBuilder setLocalLogNominalEntries(final int lgNomEntries) {
bLocalLgNomLongs = lgNomEntries;
diff --git a/src/main/java/org/apache/datasketches/tuple/AnotB.java b/src/main/java/org/apache/datasketches/tuple/AnotB.java
index ff5b37b..8dd3559 100644
--- a/src/main/java/org/apache/datasketches/tuple/AnotB.java
+++ b/src/main/java/org/apache/datasketches/tuple/AnotB.java
@@ -139,7 +139,7 @@ public final class AnotB<S extends Summary> {
final long thetaLongB = skB.getThetaLong();
thetaLong_ = Math.min(thetaLong_, thetaLongB);
//Build hashtable and removes hashes of skB >= theta
- final int countB = skB.getRetainedEntries();
+ final int countB = skB.getRetainedEntries(true);
final long[] hashTableB =
convertToHashTable(extractThetaHashArray(skB, countB), countB,
thetaLong_, REBUILD_THRESHOLD);
@@ -296,7 +296,7 @@ public final class AnotB<S extends Summary> {
//skB is not empty
final long thetaLongB = skB.getThetaLong();
final long thetaLong = Math.min(thetaLongA, thetaLongB);
- final int countB = skB.getRetainedEntries();
+ final int countB = skB.getRetainedEntries(true);
//Build/rebuild hashtable and removes hashes of skB >= thetaLong
final long[] hashTableB = //the following convert works for all theta sketches
diff --git a/src/main/java/org/apache/datasketches/tuple/Intersection.java b/src/main/java/org/apache/datasketches/tuple/Intersection.java
index a88223b..5dd5ea2 100644
--- a/src/main/java/org/apache/datasketches/tuple/Intersection.java
+++ b/src/main/java/org/apache/datasketches/tuple/Intersection.java
@@ -137,7 +137,7 @@ public class Intersection<S extends Summary> {
firstCall_ = false;
// input sketch is not null, could be first or next call
final long thetaLongIn = sketchIn.getThetaLong();
- final int countIn = sketchIn.getRetainedEntries();
+ final int countIn = sketchIn.getRetainedEntries(true);
thetaLong_ = min(thetaLong_, thetaLongIn); //Theta rule
// Empty rule extended in case incoming sketch does not have empty bit properly set
empty_ |= (countIn == 0) && (thetaLongIn == Long.MAX_VALUE);
@@ -159,7 +159,7 @@ public class Intersection<S extends Summary> {
}
final org.apache.datasketches.theta.Sketch nextSketch = sketchIn;
//Match nextSketch data with local instance data, filtering by theta
- final int maxMatchSize = min(hashTables_.count_, nextSketch.getRetainedEntries());
+ final int maxMatchSize = min(hashTables_.count_, nextSketch.getRetainedEntries(true));
final long[] matchHashArr = new long[maxMatchSize];
S[] matchSummaries = null;
@@ -263,7 +263,7 @@ public class Intersection<S extends Summary> {
}
void fromSketch(final org.apache.datasketches.theta.Sketch sketch, final S summary) {
- count_ = sketch.getRetainedEntries();
+ count_ = sketch.getRetainedEntries(true);
lgTableSize_ = getLgTableSize(count_);
S mySummary = null;
diff --git a/src/test/java/org/apache/datasketches/theta/BackwardConversions.java b/src/test/java/org/apache/datasketches/theta/BackwardConversions.java
index 5e3ed1f..0122f23 100644
--- a/src/test/java/org/apache/datasketches/theta/BackwardConversions.java
+++ b/src/test/java/org/apache/datasketches/theta/BackwardConversions.java
@@ -210,8 +210,8 @@ public class BackwardConversions {
return wmem;
}
//General CompactSketch
- final int preLongs = skV3.getCurrentPreambleLongs(true);
- final int entries = skV3.getRetainedEntries();
+ final int preLongs = skV3.getCompactPreambleLongs();
+ final int entries = skV3.getRetainedEntries(true);
final boolean unordered = !(skV3.isOrdered());
final byte flags = (byte) (0xA | (unordered ? 16 : 0)); //Unordered, NoRebuild, notEmpty, ReadOnly, LE
wmem = WritableMemory.allocate((preLongs + entries) << 3);
diff --git a/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java b/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
index 502dd4b..fe8347e 100644
--- a/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/CompactSketchTest.java
@@ -118,7 +118,7 @@ public class CompactSketchTest {
assertNotNull(testSk.iterator());
assertEquals(testSk.toByteArray().length, 8);
assertEquals(testSk.getCache().length, 0);
- assertEquals(testSk.getCurrentPreambleLongs(true), 1);
+ assertEquals(testSk.getCompactPreambleLongs(), 1);
}
private static void checkSingleItemSketch(Sketch testSk, Sketch refSk) {
@@ -136,7 +136,7 @@ public class CompactSketchTest {
assertNotNull(testSk.iterator());
assertEquals(testSk.toByteArray().length, 16);
assertEquals(testSk.getCache().length, 1);
- assertEquals(testSk.getCurrentPreambleLongs(true), 1);
+ assertEquals(testSk.getCompactPreambleLongs(), 1);
}
private static void checkOtherCompactSketch(Sketch testSk, Sketch refSk, boolean ordered) {
@@ -162,12 +162,12 @@ public class CompactSketchTest {
assertTrue(testSk instanceof HeapCompactSketch);
}
assertEquals(testSk.getSeedHash(), refSk.getSeedHash());
- assertEquals(testSk.getRetainedEntries(true), refSk.getRetainedEntries());
+ assertEquals(testSk.getRetainedEntries(true), refSk.getRetainedEntries(true));
assertEquals(testSk.getEstimate(), refSk.getEstimate(), 0.0);
assertEquals(testSk.getCurrentBytes(), refSk.getCurrentBytes());
assertEquals(testSk.toByteArray().length, refSk.toByteArray().length);
assertEquals(testSk.getCache().length, refSk.getCache().length);
- assertEquals(testSk.getCurrentPreambleLongs(true), refSk.getCurrentPreambleLongs(true));
+ assertEquals(testSk.getCompactPreambleLongs(), refSk.getCompactPreambleLongs());
}
@Test
@@ -350,7 +350,7 @@ public class CompactSketchTest {
void check(CompactSketch csk) {
assertEquals(csk.getClass().getSimpleName(), classType, "ClassType");
- assertEquals(csk.getRetainedEntries(), count, "curCount");
+ assertEquals(csk.getRetainedEntries(true), count, "curCount");
assertEquals(csk.getCurrentBytes(), bytes, "Bytes" );
assertEquals(csk.isCompact(), compact, "Compact");
assertEquals(csk.isEmpty(), empty, "Empty");
diff --git a/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java
index 4a8e836..023a326 100644
--- a/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/ConcurrentDirectQuickSelectSketchTest.java
@@ -20,6 +20,7 @@
package org.apache.datasketches.theta;
import static org.apache.datasketches.Util.DEFAULT_UPDATE_SEED;
+import static org.apache.datasketches.theta.ConcurrentHeapQuickSelectSketchTest.waitForBgPropagationToComplete;
import static org.apache.datasketches.theta.PreambleUtil.FAMILY_BYTE;
import static org.apache.datasketches.theta.PreambleUtil.LG_NOM_LONGS_BYTE;
import static org.apache.datasketches.theta.PreambleUtil.SER_VER_BYTE;
@@ -31,8 +32,8 @@ import org.apache.datasketches.Family;
import org.apache.datasketches.HashOperations;
import org.apache.datasketches.SketchesArgumentException;
import org.apache.datasketches.memory.Memory;
-import org.apache.datasketches.memory.WritableDirectHandle;
import org.apache.datasketches.memory.WritableMemory;
+import org.apache.datasketches.theta.ConcurrentHeapQuickSelectSketchTest.SharedLocal;
import org.testng.annotations.Test;
/**
@@ -40,664 +41,476 @@ import org.testng.annotations.Test;
*/
@SuppressWarnings("javadoc")
public class ConcurrentDirectQuickSelectSketchTest {
-
- private int lgK;
- private volatile UpdateSketch shared;
-
- @Test(expectedExceptions = SketchesArgumentException.class)
- public void checkBadSerVer() {
- lgK = 9;
- int k = 1 << lgK;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
-
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(mem);
- UpdateSketch local = bldr.buildLocal(shared);
-
- assertTrue(local.isEmpty());
-
- for (int i = 0; i< k; i++) { local.update(i); }
- waitForBgPropagationToComplete();
-
- assertFalse(local.isEmpty());
- assertEquals(local.getEstimate(), k, 0.0);
- assertEquals(shared.getRetainedEntries(false), k);
-
- mem.putByte(SER_VER_BYTE, (byte) 0); //corrupt the SerVer byte
-
- Sketch.wrap(mem);
- }
- }
+ private static final long SEED = DEFAULT_UPDATE_SEED;
@Test
public void checkDirectCompactConversion() {
- lgK = 9;
- int k = 1 << lgK;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
- buildSharedReturnLocalSketch(mem);
- assertTrue(shared instanceof ConcurrentDirectQuickSelectSketch);
- assertTrue(shared.compact().isCompact());
- }
- }
-
- @Test(expectedExceptions = SketchesArgumentException.class)
- public void checkConstructorKtooSmall() {
- lgK = 3;
- int k = 1 << lgK;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
- buildSharedReturnLocalSketch(mem);
- }
- }
-
- @Test(expectedExceptions = SketchesArgumentException.class)
- public void checkConstructorMemTooSmall() {
- lgK = 4;
- int k = 1 << lgK;
- try (WritableDirectHandle h = makeNativeMemory(k/2)) {
- WritableMemory mem = h.get();
- buildSharedReturnLocalSketch(mem);
- }
- }
-
- @Test(expectedExceptions = SketchesArgumentException.class)
- public void checkHeapifyIllegalFamilyID_heapify() {
- lgK = 9;
- int k = 1 << lgK;
- int bytes = (k << 4) + (Family.QUICKSELECT.getMinPreLongs() << 3);
- WritableMemory mem = WritableMemory.wrap(new byte[bytes]);
- buildSharedReturnLocalSketch(mem);
-
- mem.putByte(FAMILY_BYTE, (byte) 0); //corrupt the Family ID byte
-
- //try to heapify the corrupted mem
- Sketch.heapify(mem); //catch in Sketch.constructHeapSketch
+ int lgK = 9;
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ assertTrue(sl.shared instanceof ConcurrentDirectQuickSelectSketch);
+ assertTrue(sl.shared.compact().isCompact());
}
@Test
public void checkHeapifyMemoryEstimating() {
- lgK = 9;
+ int lgK = 9;
int k = 1 << lgK;
int u = 2*k;
- boolean estimating = (u > k);
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
-
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(mem);
- UpdateSketch local = bldr.buildLocal(shared);
- for (int i=0; i<u; i++) { local.update(i); }
- waitForBgPropagationToComplete();
-
- double sk1est = local.getEstimate();
- double sk1lb = local.getLowerBound(2);
- double sk1ub = local.getUpperBound(2);
- assertEquals(local.isEstimationMode(), estimating);
- assertEquals(local.getClass().getSimpleName(), "ConcurrentHeapThetaBuffer");
- int curCount1 = shared.getRetainedEntries(true);
- assertTrue(local.isDirect());
- assertTrue(local.hasMemory());
- assertEquals(local.getCurrentPreambleLongs(false), 3);
-
- UpdateSketch sharedHeap = Sketches.heapifyUpdateSketch(mem);
- assertEquals(sharedHeap.getEstimate(), sk1est);
- assertEquals(sharedHeap.getLowerBound(2), sk1lb);
- assertEquals(sharedHeap.getUpperBound(2), sk1ub);
- assertFalse(sharedHeap.isEmpty());
- assertEquals(sharedHeap.isEstimationMode(), estimating);
- assertEquals(sharedHeap.getClass().getSimpleName(), "HeapQuickSelectSketch");
- int curCount2 = sharedHeap.getRetainedEntries(true);
- long[] cache = sharedHeap.getCache();
- assertEquals(curCount1, curCount2);
- long thetaLong = sharedHeap.getThetaLong();
- int cacheCount = HashOperations.count(cache, thetaLong);
- assertEquals(curCount1, cacheCount);
- assertFalse(sharedHeap.isDirect());
- assertFalse(sharedHeap.hasMemory());
- }
- }
+ UpdateSketch shared = sl.shared; //off-heap
+ UpdateSketch local = sl.local;
- @Test(expectedExceptions = SketchesArgumentException.class)
- public void checkWrapIllegalFamilyID_wrap() {
- lgK = 9;
- int k = 1 << lgK;
- int maxBytes = (k << 4) + (Family.QUICKSELECT.getMinPreLongs() << 3);
- WritableMemory mem = WritableMemory.wrap(new byte[maxBytes]);
+ for (int i=0; i<u; i++) { local.update(i); }
+ waitForBgPropagationToComplete(shared);
- buildSharedReturnLocalSketch(mem);
+ assertEquals(shared.getClass().getSimpleName(), "ConcurrentDirectQuickSelectSketch");
+ assertEquals(local.getClass().getSimpleName(), "ConcurrentHeapThetaBuffer");
- mem.putByte(FAMILY_BYTE, (byte) 0); //corrupt the Sketch ID byte
+ //This sharedHeap is not linked to the concurrent local buffer
+ UpdateSketch sharedHeap = Sketches.heapifyUpdateSketch(sl.wmem);
+ assertEquals(sharedHeap.getClass().getSimpleName(), "HeapQuickSelectSketch");
- //try to wrap the corrupted mem
- Sketch.wrap(mem); //catch in Sketch.constructDirectSketch
+ checkMemoryDirectProxyMethods(local, shared);
+ checkOtherProxyMethods(local, shared);
+ checkOtherProxyMethods(local, sharedHeap);
+
+ int curCount1 = shared.getRetainedEntries(true);
+ int curCount2 = sharedHeap.getRetainedEntries(true);
+ assertEquals(curCount1, curCount2);
+ long[] cache = sharedHeap.getCache();
+ long thetaLong = sharedHeap.getThetaLong();
+ int cacheCount = HashOperations.count(cache, thetaLong);
+ assertEquals(curCount1, cacheCount);
+ assertEquals(local.getCurrentPreambleLongs(), 3);
}
- @Test(expectedExceptions = SketchesArgumentException.class)
- public void checkWrapIllegalFamilyID_direct() {
- lgK = 9;
+ @Test
+ public void checkHeapifyByteArrayExact() {
+ int lgK = 9;
int k = 1 << lgK;
- int maxBytes = (k << 4) + (Family.QUICKSELECT.getMinPreLongs() << 3);
- WritableMemory mem = WritableMemory.wrap(new byte[maxBytes]);
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
- buildSharedReturnLocalSketch(mem);
+ for (int i=0; i< k; i++) { local.update(i); }
+ waitForBgPropagationToComplete(shared);
- mem.putByte(FAMILY_BYTE, (byte) 0); //corrupt the Sketch ID byte
+ byte[] serArr = shared.toByteArray();
+ Memory srcMem = Memory.wrap(serArr);
+ Sketch recoveredShared = Sketch.heapify(srcMem);
- //try to wrap the corrupted mem
- DirectQuickSelectSketch.writableWrap(mem, DEFAULT_UPDATE_SEED);
- }
+ //reconstruct to Native/Direct
+ final int bytes = Sketch.getMaxUpdateSketchBytes(k);
+ final WritableMemory wmem = WritableMemory.allocate(bytes);
+ shared = sl.bldr.buildSharedFromSketch((UpdateSketch)recoveredShared, wmem);
+ UpdateSketch local2 = sl.bldr.buildLocal(shared);
- @Test(expectedExceptions = SketchesArgumentException.class)
- public void checkHeapifySeedConflict() {
- lgK = 9;
- int k = 1 << lgK;
- long seed1 = 1021;
- long seed2 = DEFAULT_UPDATE_SEED;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
-
- final UpdateSketchBuilder bldr = configureBuilder().setSeed(seed1);
- //must build shared first
- shared = bldr.buildShared(mem);
- byte[] serArr = shared.toByteArray();
- Memory srcMem = Memory.wrap(serArr);
- Sketch.heapify(srcMem, seed2);
- }
- }
+ assertEquals(local2.getEstimate(), k, 0.0);
+ assertEquals(local2.getLowerBound(2), k, 0.0);
+ assertEquals(local2.getUpperBound(2), k, 0.0);
+ assertEquals(local2.isEmpty(), false);
+ assertEquals(local2.isEstimationMode(), false);
+ assertEquals(recoveredShared.getClass().getSimpleName(), "HeapQuickSelectSketch");
- @Test(expectedExceptions = SketchesArgumentException.class)
- public void checkCorruptLgNomLongs() {
- lgK = 4;
- int k = 1 << lgK;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
- buildSharedReturnLocalSketch(mem);
- mem.putByte(LG_NOM_LONGS_BYTE, (byte)2); //corrupt
- Sketch.heapify(mem, DEFAULT_UPDATE_SEED);
- }
- }
+ // Run toString just to make sure that we can pull out all of the relevant information.
+ // That is, this is being run for its side-effect of accessing things.
+ // If something is wonky, it will generate an exception and fail the test.
+ local2.toString(true, true, 8, true);
- @Test(expectedExceptions = UnsupportedOperationException.class)
- public void checkIllegalHashUpdate() {
- lgK = 4;
- int k = 1 << lgK;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- buildSharedReturnLocalSketch(h.get());
- shared.hashUpdate(1);
- }
- }
-
- @Test
- public void checkHeapifyByteArrayExact() {
- lgK = 9;
- int k = 1 << lgK;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
-
- final UpdateSketchBuilder bldr = configureBuilder();
- UpdateSketch local = buildSharedReturnLocalSketch(mem);
-
- for (int i=0; i< k; i++) { local.update(i); }
- waitForBgPropagationToComplete();
-
- byte[] serArr = shared.toByteArray();
- Memory srcMem = Memory.wrap(serArr);
- Sketch recoveredShared = Sketch.heapify(srcMem);
-
- //reconstruct to Native/Direct
- final int bytes = Sketch.getMaxUpdateSketchBytes(k);
- final WritableMemory wmem = WritableMemory.allocate(bytes);
- shared = bldr.buildSharedFromSketch((UpdateSketch)recoveredShared, wmem);
- UpdateSketch local2 = bldr.buildLocal(shared);
-
- assertEquals(local2.getEstimate(), k, 0.0);
- assertEquals(local2.getLowerBound(2), k, 0.0);
- assertEquals(local2.getUpperBound(2), k, 0.0);
- assertEquals(local2.isEmpty(), false);
- assertEquals(local2.isEstimationMode(), false);
- assertEquals(recoveredShared.getClass().getSimpleName(), "HeapQuickSelectSketch");
-
- // Run toString just to make sure that we can pull out all of the relevant information.
- // That is, this is being run for its side-effect of accessing things.
- // If something is wonky, it will generate an exception and fail the test.
- local2.toString(true, true, 8, true);
- }
}
@Test
public void checkHeapifyByteArrayEstimating() {
- lgK = 12;
+ int lgK = 12;
int k = 1 << lgK;
int u = 2*k;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
- final UpdateSketchBuilder bldr = configureBuilder();
- UpdateSketch local = buildSharedReturnLocalSketch(mem);
-
- for (int i=0; i<u; i++) { local.update(i); }
- waitForBgPropagationToComplete();
-
- double uskEst = local.getEstimate();
- double uskLB = local.getLowerBound(2);
- double uskUB = local.getUpperBound(2);
- assertEquals(local.isEstimationMode(), true);
-
- byte[] serArr = shared.toByteArray();
- Memory srcMem = Memory.wrap(serArr);
- Sketch recoveredShared = Sketch.heapify(srcMem);
-
- //reconstruct to Native/Direct
- final int bytes = Sketch.getMaxUpdateSketchBytes(k);
- final WritableMemory wmem = WritableMemory.allocate(bytes);
- shared = bldr.buildSharedFromSketch((UpdateSketch)recoveredShared, wmem);
- UpdateSketch local2 = bldr.buildLocal(shared);
-
-
- assertEquals(local2.getEstimate(), uskEst);
- assertEquals(local2.getLowerBound(2), uskLB);
- assertEquals(local2.getUpperBound(2), uskUB);
- assertEquals(local2.isEmpty(), false);
- assertEquals(local2.isEstimationMode(), true);
- assertEquals(recoveredShared.getClass().getSimpleName(), "HeapQuickSelectSketch");
- }
+
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
+
+ for (int i=0; i<u; i++) { local.update(i); }
+ waitForBgPropagationToComplete(shared);
+
+ double uskEst = local.getEstimate();
+ double uskLB = local.getLowerBound(2);
+ double uskUB = local.getUpperBound(2);
+ assertEquals(local.isEstimationMode(), true);
+
+ byte[] serArr = shared.toByteArray();
+ Memory srcMem = Memory.wrap(serArr);
+ Sketch recoveredShared = Sketch.heapify(srcMem);
+
+ //reconstruct to Native/Direct
+ final int bytes = Sketch.getMaxUpdateSketchBytes(k);
+ final WritableMemory wmem = WritableMemory.allocate(bytes);
+ shared = sl.bldr.buildSharedFromSketch((UpdateSketch)recoveredShared, wmem);
+ UpdateSketch local2 = sl.bldr.buildLocal(shared);
+
+ assertEquals(local2.getEstimate(), uskEst);
+ assertEquals(local2.getLowerBound(2), uskLB);
+ assertEquals(local2.getUpperBound(2), uskUB);
+ assertEquals(local2.isEmpty(), false);
+ assertEquals(local2.isEstimationMode(), true);
+ assertEquals(recoveredShared.getClass().getSimpleName(), "HeapQuickSelectSketch");
}
@Test
public void checkWrapMemoryEst() {
- lgK = 9;
+ int lgK = 9;
int k = 1 << lgK;
int u = 2*k;
boolean estimating = (u > k);
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
- UpdateSketch local = buildSharedReturnLocalSketch(mem);
- for (int i=0; i<u; i++) { local.update(i); }
- waitForBgPropagationToComplete();
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
- double sk1est = local.getEstimate();
- double sk1lb = local.getLowerBound(2);
- double sk1ub = local.getUpperBound(2);
- assertEquals(local.isEstimationMode(), estimating);
+ for (int i=0; i<u; i++) { local.update(i); }
+ waitForBgPropagationToComplete(shared);
- Sketch local2 = Sketch.wrap(mem);
+ double sk1est = local.getEstimate();
+ double sk1lb = local.getLowerBound(2);
+ double sk1ub = local.getUpperBound(2);
+ assertEquals(local.isEstimationMode(), estimating);
- assertEquals(local2.getEstimate(), sk1est);
- assertEquals(local2.getLowerBound(2), sk1lb);
- assertEquals(local2.getUpperBound(2), sk1ub);
- assertEquals(local2.isEmpty(), false);
- assertEquals(local2.isEstimationMode(), estimating);
- }
+ Sketch local2 = Sketch.wrap(sl.wmem);
+
+ assertEquals(local2.getEstimate(), sk1est);
+ assertEquals(local2.getLowerBound(2), sk1lb);
+ assertEquals(local2.getUpperBound(2), sk1ub);
+ assertEquals(local2.isEmpty(), false);
+ assertEquals(local2.isEstimationMode(), estimating);
}
@Test
public void checkDQStoCompactForms() {
- lgK = 9;
+ int lgK = 9;
int k = 1 << lgK;
int u = 4*k;
boolean estimating = (u > k);
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
-
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(mem);
- UpdateSketch local = bldr.buildLocal(shared);
-
- assertEquals(local.getClass().getSimpleName(), "ConcurrentHeapThetaBuffer");
- assertTrue(local.isDirect());
- assertTrue(local.hasMemory());
-
- for (int i=0; i<u; i++) { local.update(i); }
- waitForBgPropagationToComplete();
-
- shared.rebuild(); //forces size back to k
-
- //get baseline values
- double localEst = local.getEstimate();
- double localLB = local.getLowerBound(2);
- double localUB = local.getUpperBound(2);
- assertEquals(local.isEstimationMode(), estimating);
-
- CompactSketch csk;
-
- csk = shared.compact(false, null);
- assertEquals(csk.getEstimate(), localEst);
- assertEquals(csk.getLowerBound(2), localLB);
- assertEquals(csk.getUpperBound(2), localUB);
- assertFalse(csk.isEmpty());
- assertEquals(csk.isEstimationMode(), estimating);
- assertEquals(csk.getClass().getSimpleName(), "HeapCompactSketch");
-
- csk = shared.compact(true, null);
- assertEquals(csk.getEstimate(), localEst);
- assertEquals(csk.getLowerBound(2), localLB);
- assertEquals(csk.getUpperBound(2), localUB);
- assertFalse(csk.isEmpty());
- assertEquals(csk.isEstimationMode(), estimating);
- assertEquals(csk.getClass().getSimpleName(), "HeapCompactSketch");
-
- int bytes = local.getCompactBytes(); //TODO WHAT IS GOING ON HERE
- assertEquals(bytes, (k*8) + (Family.COMPACT.getMaxPreLongs() << 3));
- byte[] memArr2 = new byte[bytes];
- WritableMemory mem2 = WritableMemory.wrap(memArr2);
-
- csk = shared.compact(false, mem2);
- assertEquals(csk.getEstimate(), localEst);
- assertEquals(csk.getLowerBound(2), localLB);
- assertEquals(csk.getUpperBound(2), localUB);
- assertFalse(csk.isEmpty());
- assertEquals(csk.isEstimationMode(), estimating);
- assertEquals(csk.getClass().getSimpleName(), "DirectCompactSketch");
-
- mem2.clear();
- csk = shared.compact(true, mem2);
- assertEquals(csk.getEstimate(), localEst);
- assertEquals(csk.getLowerBound(2), localLB);
- assertEquals(csk.getUpperBound(2), localUB);
- assertFalse(csk.isEmpty());
- assertEquals(csk.isEstimationMode(), estimating);
- assertEquals(csk.getClass().getSimpleName(), "DirectCompactSketch");
- csk.toString(false, true, 0, false);
- }
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
+
+ assertEquals(local.getClass().getSimpleName(), "ConcurrentHeapThetaBuffer");
+ assertFalse(local.isDirect());
+ assertTrue(local.hasMemory());
+
+ for (int i=0; i<u; i++) { local.update(i); }
+ waitForBgPropagationToComplete(shared);
+
+ shared.rebuild(); //forces size back to k
+
+ //get baseline values
+ double localEst = local.getEstimate();
+ double localLB = local.getLowerBound(2);
+ double localUB = local.getUpperBound(2);
+ assertEquals(local.isEstimationMode(), estimating);
+
+ CompactSketch csk;
+
+ csk = shared.compact(false, null);
+ assertEquals(csk.getEstimate(), localEst);
+ assertEquals(csk.getLowerBound(2), localLB);
+ assertEquals(csk.getUpperBound(2), localUB);
+ assertFalse(csk.isEmpty());
+ assertEquals(csk.isEstimationMode(), estimating);
+ assertEquals(csk.getClass().getSimpleName(), "HeapCompactSketch");
+
+ csk = shared.compact(true, null);
+ assertEquals(csk.getEstimate(), localEst);
+ assertEquals(csk.getLowerBound(2), localLB);
+ assertEquals(csk.getUpperBound(2), localUB);
+ assertFalse(csk.isEmpty());
+ assertEquals(csk.isEstimationMode(), estimating);
+ assertEquals(csk.getClass().getSimpleName(), "HeapCompactSketch");
+
+ int bytes = shared.getCompactBytes();
+ assertEquals(bytes, (k*8) + (Family.COMPACT.getMaxPreLongs() << 3));
+ byte[] memArr2 = new byte[bytes];
+ WritableMemory mem2 = WritableMemory.wrap(memArr2);
+
+ csk = shared.compact(false, mem2);
+ assertEquals(csk.getEstimate(), localEst);
+ assertEquals(csk.getLowerBound(2), localLB);
+ assertEquals(csk.getUpperBound(2), localUB);
+ assertFalse(csk.isEmpty());
+ assertEquals(csk.isEstimationMode(), estimating);
+ assertEquals(csk.getClass().getSimpleName(), "DirectCompactSketch");
+
+ mem2.clear();
+ csk = shared.compact(true, mem2);
+ assertEquals(csk.getEstimate(), localEst);
+ assertEquals(csk.getLowerBound(2), localLB);
+ assertEquals(csk.getUpperBound(2), localUB);
+ assertFalse(csk.isEmpty());
+ assertEquals(csk.isEstimationMode(), estimating);
+ assertEquals(csk.getClass().getSimpleName(), "DirectCompactSketch");
+ csk.toString(false, true, 0, false);
}
@Test
public void checkDQStoCompactEmptyForms() {
- lgK = 9;
- int k = 1 << lgK;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
-
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(mem);
- UpdateSketch local = bldr.buildLocal(shared);
-
- //empty
- local.toString(false, true, 0, false); //exercise toString
- assertEquals(local.getClass().getSimpleName(), "ConcurrentHeapThetaBuffer");
- double localEst = local.getEstimate();
- double localLB = local.getLowerBound(2);
- double localUB = local.getUpperBound(2);
- assertFalse(local.isEstimationMode());
-
- int bytes = local.getCompactBytes(); //compact form
- assertEquals(bytes, 8);
- byte[] memArr2 = new byte[bytes];
- WritableMemory mem2 = WritableMemory.wrap(memArr2);
-
- CompactSketch csk2 = shared.compact(false, mem2);
- assertEquals(csk2.getEstimate(), localEst);
- assertEquals(csk2.getLowerBound(2), localLB);
- assertEquals(csk2.getUpperBound(2), localUB);
- assertTrue(csk2.isEmpty());
- assertFalse(csk2.isEstimationMode());
- assertTrue(csk2.isOrdered());
- CompactSketch csk3 = shared.compact(true, mem2);
- csk3.toString(false, true, 0, false);
- csk3.toString();
- assertEquals(csk3.getEstimate(), localEst);
- assertEquals(csk3.getLowerBound(2), localLB);
- assertEquals(csk3.getUpperBound(2), localUB);
- assertTrue(csk3.isEmpty());
- assertFalse(csk3.isEstimationMode());
- assertTrue(csk2.isOrdered());
- }
+ int lgK = 9;
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
+
+ //empty
+ local.toString(false, true, 0, false); //exercise toString
+ assertEquals(local.getClass().getSimpleName(), "ConcurrentHeapThetaBuffer");
+ double localEst = local.getEstimate();
+ double localLB = local.getLowerBound(2);
+ double localUB = local.getUpperBound(2);
+ assertFalse(local.isEstimationMode());
+
+ int bytes = local.getCompactBytes(); //compact form
+ assertEquals(bytes, 8);
+ byte[] memArr2 = new byte[bytes];
+ WritableMemory mem2 = WritableMemory.wrap(memArr2);
+
+ CompactSketch csk2 = shared.compact(false, mem2);
+ assertEquals(csk2.getEstimate(), localEst);
+ assertEquals(csk2.getLowerBound(2), localLB);
+ assertEquals(csk2.getUpperBound(2), localUB);
+ assertTrue(csk2.isEmpty());
+ assertFalse(csk2.isEstimationMode());
+ assertTrue(csk2.isOrdered());
+ CompactSketch csk3 = shared.compact(true, mem2);
+ csk3.toString(false, true, 0, false);
+ csk3.toString();
+ assertEquals(csk3.getEstimate(), localEst);
+ assertEquals(csk3.getLowerBound(2), localLB);
+ assertEquals(csk3.getUpperBound(2), localUB);
+ assertTrue(csk3.isEmpty());
+ assertFalse(csk3.isEstimationMode());
+ assertTrue(csk2.isOrdered());
}
@Test
public void checkEstMode() {
- lgK = 12;
+ int lgK = 12;
int k = 1 << lgK;
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
-
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(mem);
- UpdateSketch local = bldr.buildLocal(shared);
-
- assertTrue(local.isEmpty());
- int u = 3*k;
-
- for (int i = 0; i< u; i++) { local.update(i); }
- waitForBgPropagationToComplete();
+ assertTrue(local.isEmpty());
+ int u = 3*k;
- assertTrue(shared.getRetainedEntries(false) > k);
- }
+ for (int i = 0; i< u; i++) { local.update(i); }
+ waitForBgPropagationToComplete(shared);
+ assertTrue(shared.getRetainedEntries(false) > k);
}
@Test
public void checkErrorBounds() {
- lgK = 9;
+ int lgK = 9;
int k = 1 << lgK;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
-
- UpdateSketch local = buildSharedReturnLocalSketch(mem);
-
- //Exact mode
- for (int i = 0; i < k; i++ ) { local.update(i); }
- waitForBgPropagationToComplete();
-
- double est = local.getEstimate();
- double lb = local.getLowerBound(2);
- double ub = local.getUpperBound(2);
- assertEquals(est, ub, 0.0);
- assertEquals(est, lb, 0.0);
-
- //Est mode
- int u = 100*k;
- for (int i = k; i < u; i++ ) {
- local.update(i);
- local.update(i); //test duplicate rejection
- }
- waitForBgPropagationToComplete();
- est = local.getEstimate();
- lb = local.getLowerBound(2);
- ub = local.getUpperBound(2);
- assertTrue(est <= ub);
- assertTrue(est >= lb);
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
+
+ //Exact mode
+ for (int i = 0; i < k; i++ ) { local.update(i); }
+ waitForBgPropagationToComplete(shared);
+
+ double est = local.getEstimate();
+ double lb = local.getLowerBound(2);
+ double ub = local.getUpperBound(2);
+ assertEquals(est, ub, 0.0);
+ assertEquals(est, lb, 0.0);
+
+ //Est mode
+ int u = 100*k;
+ for (int i = k; i < u; i++ ) {
+ local.update(i);
+ local.update(i); //test duplicate rejection
}
+ waitForBgPropagationToComplete(shared);
+ est = local.getEstimate();
+ lb = local.getLowerBound(2);
+ ub = local.getUpperBound(2);
+ assertTrue(est <= ub);
+ assertTrue(est >= lb);
}
@Test
public void checkUpperAndLowerBounds() {
- lgK = 9;
+ int lgK = 9;
int k = 1 << lgK;
int u = 2*k;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
-
- UpdateSketch local = buildSharedReturnLocalSketch(mem);
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
- for (int i = 0; i < u; i++ ) { local.update(i); }
- waitForBgPropagationToComplete();
+ for (int i = 0; i < u; i++ ) { local.update(i); }
+ waitForBgPropagationToComplete(shared);
- double est = local.getEstimate();
- double ub = local.getUpperBound(1);
- double lb = local.getLowerBound(1);
- assertTrue(ub > est);
- assertTrue(lb < est);
- }
+ double est = local.getEstimate();
+ double ub = local.getUpperBound(1);
+ double lb = local.getLowerBound(1);
+ assertTrue(ub > est);
+ assertTrue(lb < est);
}
@Test
public void checkRebuild() {
- lgK = 9;
+ int lgK = 9;
int k = 1 << lgK;
int u = 4*k;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(mem);
- UpdateSketch local = bldr.buildLocal(shared);
+ assertTrue(local.isEmpty());
- assertTrue(local.isEmpty());
+ for (int i = 0; i< u; i++) { local.update(i); }
+ waitForBgPropagationToComplete(shared);
- for (int i = 0; i< u; i++) { local.update(i); }
- waitForBgPropagationToComplete();
+ assertFalse(local.isEmpty());
+ assertTrue(local.getEstimate() > 0.0);
+ assertTrue(shared.getRetainedEntries(false) >= k);
- assertFalse(local.isEmpty());
- assertTrue(local.getEstimate() > 0.0);
- assertTrue(shared.getRetainedEntries(false) >= k);
-
- shared.rebuild();
- assertEquals(shared.getRetainedEntries(false), k);
- assertEquals(shared.getRetainedEntries(true), k);
- local.rebuild();
- assertEquals(shared.getRetainedEntries(false), k);
- assertEquals(shared.getRetainedEntries(true), k);
- }
+ shared.rebuild();
+ assertEquals(shared.getRetainedEntries(false), k);
+ assertEquals(shared.getRetainedEntries(true), k);
+ local.rebuild();
+ assertEquals(shared.getRetainedEntries(false), k);
+ assertEquals(shared.getRetainedEntries(true), k);
}
@Test
public void checkResetAndStartingSubMultiple() {
- lgK = 9;
+ int lgK = 9;
int k = 1 << lgK;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
-
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(mem);
- UpdateSketch local = bldr.buildLocal(shared);
-
- assertTrue(local.isEmpty());
-
- int u = 4*k;
- for (int i = 0; i< u; i++) { local.update(i); }
- waitForBgPropagationToComplete();
-
- assertFalse(local.isEmpty());
- assertTrue(shared.getRetainedEntries(false) >= k);
- assertTrue(local.getThetaLong() < Long.MAX_VALUE);
-
- shared.reset();
- local.reset();
- assertTrue(local.isEmpty());
- assertEquals(shared.getRetainedEntries(false), 0);
- assertEquals(local.getEstimate(), 0.0, 0.0);
- assertEquals(local.getThetaLong(), Long.MAX_VALUE);
- }
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
+
+ assertTrue(local.isEmpty());
+
+ int u = 4*k;
+ for (int i = 0; i< u; i++) { local.update(i); }
+ waitForBgPropagationToComplete(shared);
+
+ assertFalse(local.isEmpty());
+ assertTrue(shared.getRetainedEntries(false) >= k);
+ assertTrue(local.getThetaLong() < Long.MAX_VALUE);
+
+ shared.reset();
+ local.reset();
+ assertTrue(local.isEmpty());
+ assertEquals(shared.getRetainedEntries(false), 0);
+ assertEquals(local.getEstimate(), 0.0, 0.0);
+ assertEquals(local.getThetaLong(), Long.MAX_VALUE);
}
@Test
public void checkExactModeMemoryArr() {
- lgK = 12;
+ int lgK = 12;
int k = 1 << lgK;
int u = k;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
-
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(mem);
- UpdateSketch local = bldr.buildLocal(shared);
- assertTrue(local.isEmpty());
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
+ assertTrue(local.isEmpty());
- for (int i = 0; i< u; i++) { local.update(i); }
- waitForBgPropagationToComplete();
+ for (int i = 0; i< u; i++) { local.update(i); }
+ waitForBgPropagationToComplete(shared);
- assertEquals(local.getEstimate(), u, 0.0);
- assertEquals(shared.getRetainedEntries(false), u);
- }
+ assertEquals(local.getEstimate(), u, 0.0);
+ assertEquals(shared.getRetainedEntries(false), u);
}
@Test
public void checkEstModeMemoryArr() {
- lgK = 12;
+ int lgK = 12;
int k = 1 << lgK;
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
+ assertTrue(local.isEmpty());
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
-
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(mem);
- UpdateSketch local = bldr.buildLocal(shared);
- assertTrue(local.isEmpty());
-
- int u = 3*k;
- for (int i = 0; i< u; i++) { local.update(i); }
- waitForBgPropagationToComplete();
+ int u = 3*k;
+ for (int i = 0; i< u; i++) { local.update(i); }
+ waitForBgPropagationToComplete(shared);
- double est = local.getEstimate();
- assertTrue((est < (u * 1.05)) && (est > (u * 0.95)));
- assertTrue(shared.getRetainedEntries(false) >= k);
- }
+ double est = local.getEstimate();
+ assertTrue((est < (u * 1.05)) && (est > (u * 0.95)));
+ assertTrue(shared.getRetainedEntries(false) >= k);
}
@Test
public void checkEstModeNativeMemory() {
- lgK = 12;
+ int lgK = 12;
int k = 1 << lgK;
- int memCapacity = (k << 4) + (Family.QUICKSELECT.getMinPreLongs() << 3);
-
- try(WritableDirectHandle memHandler = WritableMemory.allocateDirect(memCapacity)) {
-
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(memHandler.get());
- UpdateSketch local = bldr.buildLocal(shared);
- assertTrue(local.isEmpty());
- int u = 3*k;
-
- for (int i = 0; i< u; i++) { local.update(i); }
- waitForBgPropagationToComplete();
- double est = local.getEstimate();
- assertTrue((est < (u * 1.05)) && (est > (u * 0.95)));
- assertTrue(shared.getRetainedEntries(false) >= k);
- }
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
+ assertTrue(local.isEmpty());
+
+ int u = 3*k;
+ for (int i = 0; i< u; i++) { local.update(i); }
+ waitForBgPropagationToComplete(shared);
+ double est = local.getEstimate();
+ assertTrue((est < (u * 1.05)) && (est > (u * 0.95)));
+ assertTrue(shared.getRetainedEntries(false) >= k);
}
@Test
public void checkConstructReconstructFromMemory() {
- lgK = 12;
+ int lgK = 12;
int k = 1 << lgK;
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(h.get());
- UpdateSketch local = bldr.buildLocal(shared);
- assertTrue(local.isEmpty());
- int u = 3*k;
-
- for (int i = 0; i< u; i++) { local.update(i); } //force estimation
- waitForBgPropagationToComplete();
-
- double est1 = local.getEstimate();
- int count1 = shared.getRetainedEntries(false);
- assertTrue((est1 < (u * 1.05)) && (est1 > (u * 0.95)));
- assertTrue(count1 >= k);
-
- byte[] serArr;
- double est2;
-
- serArr = shared.toByteArray();
- WritableMemory mem = WritableMemory.wrap(serArr);
- UpdateSketch recoveredShared = Sketches.wrapUpdateSketch(mem);
-
- //reconstruct to Native/Direct
- final int bytes = Sketch.getMaxUpdateSketchBytes(k);
- final WritableMemory wmem = WritableMemory.allocate(bytes);
- shared = bldr.buildSharedFromSketch(recoveredShared, wmem);
- UpdateSketch local2 = bldr.buildLocal(shared);
- est2 = local2.getEstimate();
-
- assertEquals(est2, est1, 0.0);
- }
+ assertTrue(local.isEmpty());
+ int u = 3*k;
+
+ for (int i = 0; i< u; i++) { local.update(i); } //force estimation
+ waitForBgPropagationToComplete(shared);
+
+ double est1 = local.getEstimate();
+ int count1 = shared.getRetainedEntries(false);
+ assertTrue((est1 < (u * 1.05)) && (est1 > (u * 0.95)));
+ assertTrue(count1 >= k);
+
+ byte[] serArr;
+ double est2;
+
+ serArr = shared.toByteArray();
+ WritableMemory mem = WritableMemory.wrap(serArr);
+ UpdateSketch recoveredShared = Sketches.wrapUpdateSketch(mem);
+
+ //reconstruct to Native/Direct
+ final int bytes = Sketch.getMaxUpdateSketchBytes(k);
+ final WritableMemory wmem = WritableMemory.allocate(bytes);
+ shared = sl.bldr.buildSharedFromSketch(recoveredShared, wmem);
+ UpdateSketch local2 = sl.bldr.buildLocal(shared);
+ est2 = local2.getEstimate();
+
+ assertEquals(est2, est1, 0.0);
}
@Test
@@ -706,73 +519,171 @@ public class ConcurrentDirectQuickSelectSketchTest {
final UpdateSketch sk = bldr.build();
for (int i = 0; i < 1000; i++) { sk.update(i); }
final UpdateSketch shared = bldr.buildSharedFromSketch(sk, null);
- assertEquals(shared.getRetainedEntries(), 1000);
+ assertEquals(shared.getRetainedEntries(true), 1000);
assertFalse(shared.hasMemory());
}
//checks Alex's bug where lgArrLongs > lgNomLongs +1.
@Test
public void checkResizeInBigMem() {
- lgK = 14;
- int k = 1 << lgK;
+ int lgK = 14;
int u = 1 << 20;
- WritableMemory mem = WritableMemory.wrap(new byte[(8*k*16) +24]);
- UpdateSketch local = buildSharedReturnLocalSketch(mem);
- for (int i=0; i<u; i++) { local.update(i); }
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, SEED, useMem, true, 8); //mem is 8X larger than needed
+ UpdateSketch local = sl.local;
+
+ for (int i = 0; i < u; i++) { local.update(i); }
+ }
+
+ @SuppressWarnings("unused")
+ @Test(expectedExceptions = SketchesArgumentException.class)
+ public void checkConstructorKtooSmall() {
+ int lgK = 3;
+ boolean useMem = true;
+ new SharedLocal(lgK, lgK, useMem);
+ }
+
+ @Test(expectedExceptions = SketchesArgumentException.class)
+ public void checkConstructorMemTooSmall() {
+ int lgK = 4;
+ int k = 1 << lgK;
+ WritableMemory wmem = WritableMemory.allocate(k/2);
+ UpdateSketchBuilder bldr = new UpdateSketchBuilder();
+ bldr.setLogNominalEntries(lgK);
+ bldr.buildShared(wmem);
+ }
+
+ @Test(expectedExceptions = SketchesArgumentException.class)
+ public void checkHeapifyIllegalFamilyID_heapify() {
+ int lgK = 9;
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ sl.wmem.putByte(FAMILY_BYTE, (byte) 0); //corrupt the Family ID byte
+ //try to heapify the corrupted mem
+ Sketch.heapify(sl.wmem); //catch in Sketch.constructHeapSketch
}
@Test(expectedExceptions = SketchesArgumentException.class)
public void checkBadLgNomLongs() {
- int k = 16;
- lgK = 4;
- WritableMemory mem = WritableMemory.wrap(new byte[(k*16) +24]);
- buildSharedReturnLocalSketch(mem);
- mem.putByte(LG_NOM_LONGS_BYTE, (byte) 3); //Corrupt LgNomLongs byte
- DirectQuickSelectSketch.writableWrap(mem, DEFAULT_UPDATE_SEED);
+ int lgK = 4;
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ sl.wmem.putByte(LG_NOM_LONGS_BYTE, (byte) 3); //Corrupt LgNomLongs byte
+ DirectQuickSelectSketch.writableWrap(sl.wmem, DEFAULT_UPDATE_SEED);
}
@Test
public void checkBackgroundPropagation() {
- lgK = 4;
+ int lgK = 4;
int k = 1 << lgK;
int u = 10*k;
- try (WritableDirectHandle h = makeNativeMemory(k)) {
- WritableMemory mem = h.get();
-
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(mem);
- UpdateSketch local = bldr.buildLocal(shared);
- ConcurrentHeapThetaBuffer sk1 = (ConcurrentHeapThetaBuffer)local; //for internal checks
-
- assertTrue(local.isEmpty());
-
- int i = 0;
- for (; i< k; i++) {
- local.update(i);
- }
-// waitForBgPropagationToComplete();
- assertFalse(local.isEmpty());
- assertTrue(local.getEstimate() > 0.0);
- long theta1 = ((ConcurrentSharedThetaSketch)shared).getVolatileTheta();
-
- for (; i< u; i++) {
- local.update(i);
- }
- waitForBgPropagationToComplete();
-
- long theta2 = ((ConcurrentSharedThetaSketch)shared).getVolatileTheta();
- int entries = shared.getRetainedEntries(false);
- assertTrue((entries > k) || (theta2 < theta1),
- "entries="+entries+" k="+k+" theta1="+theta1+" theta2="+theta2);
-
- shared.rebuild();
- assertEquals(shared.getRetainedEntries(false), k);
- assertEquals(shared.getRetainedEntries(true), k);
- sk1.rebuild();
- assertEquals(shared.getRetainedEntries(false), k);
- assertEquals(shared.getRetainedEntries(true), k);
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
+ assertTrue(local.isEmpty());
+ ConcurrentHeapThetaBuffer sk1 = (ConcurrentHeapThetaBuffer)local; //for internal checks
+
+ int i = 0;
+ for (; i< k; i++) {
+ local.update(i);
+ }
+ waitForBgPropagationToComplete(shared);
+ assertFalse(local.isEmpty());
+ assertTrue(local.getEstimate() > 0.0);
+ long theta1 = ((ConcurrentSharedThetaSketch)shared).getVolatileTheta();
+
+ for (; i< u; i++) {
+ local.update(i);
}
+ waitForBgPropagationToComplete(shared);
+
+ long theta2 = ((ConcurrentSharedThetaSketch)shared).getVolatileTheta();
+ int entries = shared.getRetainedEntries(false);
+ assertTrue((entries > k) || (theta2 < theta1),
+ "entries="+entries+" k="+k+" theta1="+theta1+" theta2="+theta2);
+
+ shared.rebuild();
+ assertEquals(shared.getRetainedEntries(false), k);
+ assertEquals(shared.getRetainedEntries(true), k);
+ sk1.rebuild();
+ assertEquals(shared.getRetainedEntries(false), k);
+ assertEquals(shared.getRetainedEntries(true), k);
+ }
+
+ @Test(expectedExceptions = SketchesArgumentException.class)
+ public void checkBadSerVer() {
+ int lgK = 9;
+ int k = 1 << lgK;
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
+ assertTrue(local.isEmpty());
+
+ for (int i = 0; i< k; i++) { local.update(i); }
+ waitForBgPropagationToComplete(shared);
+
+ assertFalse(local.isEmpty());
+ assertEquals(local.getEstimate(), k, 0.0);
+ assertEquals(shared.getRetainedEntries(false), k);
+
+ sl.wmem.putByte(SER_VER_BYTE, (byte) 0); //corrupt the SerVer byte
+ Sketch.wrap(sl.wmem);
+ }
+
+ @Test(expectedExceptions = SketchesArgumentException.class)
+ public void checkWrapIllegalFamilyID_wrap() {
+ int lgK = 9;
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+
+ sl.wmem.putByte(FAMILY_BYTE, (byte) 0); //corrupt the Sketch ID byte
+ //try to wrap the corrupted mem
+ Sketch.wrap(sl.wmem); //catch in Sketch.constructDirectSketch
+ }
+
+ @Test(expectedExceptions = SketchesArgumentException.class)
+ public void checkWrapIllegalFamilyID_direct() {
+ int lgK = 9;
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+
+ sl.wmem.putByte(FAMILY_BYTE, (byte) 0); //corrupt the Sketch ID byte
+ //try to wrap the corrupted mem
+ DirectQuickSelectSketch.writableWrap(sl.wmem, DEFAULT_UPDATE_SEED);
+ }
+
+ @Test(expectedExceptions = SketchesArgumentException.class)
+ public void checkHeapifySeedConflict() {
+ int lgK = 9;
+ long seed1 = 1021;
+ long seed2 = DEFAULT_UPDATE_SEED;
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, seed1, useMem, true, 1);
+ UpdateSketch shared = sl.shared;
+
+ Memory srcMem = Memory.wrap(shared.toByteArray());
+ Sketch.heapify(srcMem, seed2);
+ }
+
+ @Test(expectedExceptions = SketchesArgumentException.class)
+ public void checkCorruptLgNomLongs() {
+ int lgK = 4;
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+
+ sl.wmem.putByte(LG_NOM_LONGS_BYTE, (byte)2); //corrupt
+ Sketch.heapify(sl.wmem, DEFAULT_UPDATE_SEED);
+ }
+
+ @Test(expectedExceptions = UnsupportedOperationException.class)
+ public void checkIllegalHashUpdate() {
+ int lgK = 4;
+ boolean useMem = true;
+ SharedLocal sl = new SharedLocal(lgK, lgK, useMem);
+ UpdateSketch shared = sl.shared;
+ shared.hashUpdate(1);
}
@Test
@@ -787,39 +698,20 @@ public class ConcurrentDirectQuickSelectSketchTest {
//System.out.println(s); //disable here
}
- private static final int getMaxBytes(int k) {
- return (k << 4) + (Family.QUICKSELECT.getMinPreLongs() << 3);
+ private static void checkMemoryDirectProxyMethods(Sketch local, Sketch shared) {
+ assertEquals(local.hasMemory(), shared.hasMemory());
+ assertEquals(local.isDirect(), shared.isDirect());
}
- private static WritableDirectHandle makeNativeMemory(int k) {
- return WritableMemory.allocateDirect(getMaxBytes(k));
- }
-
- private UpdateSketch buildSharedReturnLocalSketch(WritableMemory mem) {
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(mem);
- return bldr.buildLocal(shared);
- }
-
- //configures builder for both local and shared
- private UpdateSketchBuilder configureBuilder() {
- final UpdateSketchBuilder bldr = new UpdateSketchBuilder();
- bldr.setLogNominalEntries(lgK);
- bldr.setLocalLogNominalEntries(lgK);
- bldr.setSeed(DEFAULT_UPDATE_SEED);
- return bldr;
- }
-
- private void waitForBgPropagationToComplete() {
- try {
- Thread.sleep(10);
- } catch (InterruptedException e) {
- e.printStackTrace();
- }
- ((ConcurrentSharedThetaSketch)shared).awaitBgPropagationTermination();
- ConcurrentPropagationService.resetExecutorService(Thread.currentThread().getId());
- ((ConcurrentSharedThetaSketch)shared).initBgPropagationService();
+ //Does not check hasMemory(), isDirect()
+ private static void checkOtherProxyMethods(Sketch local, Sketch shared) {
+ assertEquals(local.getCompactBytes(), shared.getCompactBytes());
+ assertEquals(local.getCurrentBytes(), shared.getCurrentBytes());
+ assertEquals(local.getEstimate(), shared.getEstimate());
+ assertEquals(local.getLowerBound(2), shared.getLowerBound(2));
+ assertEquals(local.getUpperBound(2), shared.getUpperBound(2));
+ assertEquals(local.isEmpty(), shared.isEmpty());
+ assertEquals(local.isEstimationMode(), shared.isEstimationMode());
}
}
diff --git a/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java
index e119dc5..8e03aa3 100644
--- a/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/ConcurrentHeapQuickSelectSketchTest.java
@@ -32,10 +32,8 @@ import java.util.Arrays;
import org.apache.datasketches.Family;
import org.apache.datasketches.SketchesArgumentException;
-import org.apache.datasketches.SketchesStateException;
import org.apache.datasketches.memory.Memory;
import org.apache.datasketches.memory.WritableMemory;
-import org.testng.annotations.AfterMethod;
import org.testng.annotations.Test;
/**
@@ -43,28 +41,23 @@ import org.testng.annotations.Test;
*/
@SuppressWarnings("javadoc")
public class ConcurrentHeapQuickSelectSketchTest {
- private int lgK;
- private long seed = DEFAULT_UPDATE_SEED;
- //private volatile ConcurrentSharedThetaSketch shared;
- private volatile UpdateSketch shared;
+
@Test(expectedExceptions = SketchesArgumentException.class)
public void checkBadSerVer() {
- int k = 512;
- lgK = 9;
+ int lgK = 9;
+ int k = 1 << lgK;
int u = k;
- seed = DEFAULT_UPDATE_SEED;
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(null);
- UpdateSketch local = bldr.buildLocal(shared);
+ SharedLocal sl = new SharedLocal(lgK);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
assertTrue(local.isEmpty());
for (int i = 0; i< u; i++) {
local.update(i);
}
- waitForBgPropagationToComplete();
+ waitForBgPropagationToComplete(shared);
assertFalse(local.isEmpty());
assertEquals(local.getEstimate(), u, 0.0);
@@ -72,46 +65,42 @@ public class ConcurrentHeapQuickSelectSketchTest {
byte[] serArr = shared.toByteArray();
WritableMemory mem = WritableMemory.wrap(serArr);
- Sketch sk = Sketch.heapify(mem, seed);
+ Sketch sk = Sketch.heapify(mem, sl.seed);
assertTrue(sk instanceof HeapQuickSelectSketch); //Intentional promotion to Parent
mem.putByte(SER_VER_BYTE, (byte) 0); //corrupt the SerVer byte
- Sketch.heapify(mem, seed);
+ Sketch.heapify(mem, sl.seed);
}
@Test
public void checkPropagationNotOrdered() {
- int k = 256;
- lgK = 8;
+ int lgK = 8;
+ int k = 1 << lgK;
int u = 200*k;
- seed = DEFAULT_UPDATE_SEED;
- final UpdateSketchBuilder bldr = configureBuilderNotOrdered();
- assertFalse((1 << bldr.getLocalLgNominalEntries()) == 0);
- //must build shared first
- shared = bldr.buildShared(null);
- UpdateSketch local = bldr.buildLocal(shared);
-
+ SharedLocal sl = new SharedLocal(lgK, 4, false, false);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
+ assertEquals((sl.bldr.getLocalLgNominalEntries()), 4);
assertTrue(local.isEmpty());
for (int i = 0; i < u; i++) {
local.update(i);
}
- waitForBgPropagationToComplete();
+ waitForBgPropagationToComplete(shared);
assertFalse(local.isEmpty());
- assertTrue(shared.getRetainedEntries(false) <= u);
+ assertTrue(shared.getRetainedEntries(true) <= u);
}
@Test(expectedExceptions = SketchesArgumentException.class)
public void checkIllegalSketchID_UpdateSketch() {
- int k = 512;
- lgK = 9;
+ int lgK = 9;
+ int k = 1 << lgK;
int u = k;
- seed = DEFAULT_UPDATE_SEED;
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(null);
- UpdateSketch local = bldr.buildLocal(shared);
+ SharedLocal sl = new SharedLocal(lgK);
+
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
assertTrue(local.isEmpty());
assertTrue(shared instanceof ConcurrentHeapQuickSelectSketch);
for (int i = 0; i< u; i++) {
@@ -127,25 +116,25 @@ public class ConcurrentHeapQuickSelectSketchTest {
mem.putByte(FAMILY_BYTE, (byte) 0); //corrupt the Sketch ID byte
//try to heapify the corruped mem
- Sketch.heapify(mem, seed);
+ Sketch.heapify(mem, sl.seed);
}
@Test(expectedExceptions = SketchesArgumentException.class)
public void checkHeapifySeedConflict() {
- lgK = 9;
- seed = 1021;
+ int lgK = 9;
+ long seed = 1021;
long seed2 = DEFAULT_UPDATE_SEED;
- buildSharedReturnLocalSketch();
- byte[] byteArray = shared.toByteArray();
+ SharedLocal sl = new SharedLocal(lgK, lgK, seed);
+ byte[] byteArray = sl.shared.toByteArray();
Memory srcMem = Memory.wrap(byteArray);
Sketch.heapify(srcMem, seed2);
}
@Test(expectedExceptions = SketchesArgumentException.class)
public void checkHeapifyCorruptLgNomLongs() {
- lgK = 4;
- buildSharedReturnLocalSketch();
- byte[] serArr = shared.toByteArray();
+ int lgK = 4;
+ SharedLocal sl = new SharedLocal(lgK);
+ byte[] serArr = sl.shared.toByteArray();
WritableMemory srcMem = WritableMemory.wrap(serArr);
srcMem.putByte(LG_NOM_LONGS_BYTE, (byte)2); //corrupt
Sketch.heapify(srcMem, DEFAULT_UPDATE_SEED);
@@ -153,24 +142,24 @@ public class ConcurrentHeapQuickSelectSketchTest {
@Test(expectedExceptions = UnsupportedOperationException.class)
public void checkIllegalHashUpdate() {
- lgK = 4;
- buildSharedReturnLocalSketch();
- shared.hashUpdate(1);
+ int lgK = 4;
+ SharedLocal sl = new SharedLocal(lgK);
+ sl.shared.hashUpdate(1);
}
@Test
public void checkHeapifyByteArrayExact() {
- int k = 512;
- lgK = 9;
+ int lgK = 9;
+ int k = 1 << lgK;
int u = k;
- seed = DEFAULT_UPDATE_SEED;
- final UpdateSketchBuilder bldr = configureBuilder();
- UpdateSketch local = buildSharedReturnLocalSketch();
+ SharedLocal sl = new SharedLocal(lgK);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
for (int i=0; i<u; i++) {
local.update(i);
}
- waitForBgPropagationToComplete();
+ waitForBgPropagationToComplete(shared);
byte[] serArr = shared.toByteArray();
Memory srcMem = Memory.wrap(serArr);
@@ -179,8 +168,8 @@ public class ConcurrentHeapQuickSelectSketchTest {
//reconstruct to Native/Direct
final int bytes = Sketch.getMaxUpdateSketchBytes(k);
final WritableMemory wmem = WritableMemory.allocate(bytes);
- shared = bldr.buildSharedFromSketch((UpdateSketch)recoveredShared, wmem);
- UpdateSketch local2 = bldr.buildLocal(shared);
+ shared = sl.bldr.buildSharedFromSketch((UpdateSketch)recoveredShared, wmem);
+ UpdateSketch local2 = sl.bldr.buildLocal(shared);
assertEquals(local2.getEstimate(), u, 0.0);
assertEquals(local2.getLowerBound(2), u, 0.0);
@@ -193,18 +182,18 @@ public class ConcurrentHeapQuickSelectSketchTest {
@Test
public void checkHeapifyByteArrayEstimating() {
- int k = 4096;
- lgK = 12;
+ int lgK = 12;
+ int k = 1 << lgK;
int u = 2*k;
- seed = DEFAULT_UPDATE_SEED;
- final UpdateSketchBuilder bldr = configureBuilder();
- UpdateSketch local = buildSharedReturnLocalSketch();
+ SharedLocal sl = new SharedLocal(lgK);
+ UpdateSketch local = sl.local;
+ UpdateSketch shared = sl.shared;
for (int i=0; i<u; i++) {
local.update(i);
}
- waitForBgPropagationToComplete();
+ waitForBgPropagationToComplete(shared);
double localEst = local.getEstimate();
double localLB = local.getLowerBound(2);
@@ -213,12 +202,12 @@ public class ConcurrentHeapQuickSelectSketchTest {
byte[] serArr = shared.toByteArray();
Memory srcMem = Memory.wrap(serArr);
- UpdateSketch recoveredShared = UpdateSketch.heapify(srcMem, seed);
+ UpdateSketch recoveredShared = UpdateSketch.heapify(srcMem, sl.seed);
final int bytes = Sketch.getMaxUpdateSketchBytes(k);
final WritableMemory wmem = WritableMemory.allocate(bytes);
- shared = bldr.buildSharedFromSketch(recoveredShared, wmem);
- UpdateSketch local2 = bldr.buildLocal(shared);
+ shared = sl.bldr.buildSharedFromSketch(recoveredShared, wmem);
+ UpdateSketch local2 = sl.bldr.buildLocal(shared);
assertEquals(local2.getEstimate(), localEst);
assertEquals(local2.getLowerBound(2), localLB);
assertEquals(local2.getUpperBound(2), localUB);
@@ -229,18 +218,19 @@ public class ConcurrentHeapQuickSelectSketchTest {
@Test
public void checkHeapifyMemoryEstimating() {
- int k = 512;
- lgK = 9;
+ int lgK = 9;
+ int k = 1 << lgK;
int u = 2*k;
- seed = DEFAULT_UPDATE_SEED;
boolean estimating = (u > k);
- final UpdateSketchBuilder bldr = configureBuilder();
- UpdateSketch local = buildSharedReturnLocalSketch();
+
+ SharedLocal sl = new SharedLocal(lgK);
+ UpdateSketch local = sl.local;
+ UpdateSketch shared = sl.shared;
for (int i=0; i<u; i++) {
local.update(i);
}
- waitForBgPropagationToComplete();
+ waitForBgPropagationToComplete(shared);
double localEst = local.getEstimate();
double localLB = local.getLowerBound(2);
@@ -256,8 +246,8 @@ public class ConcurrentHeapQuickSelectSketchTest {
final int bytes = Sketch.getMaxUpdateSketchBytes(k);
final WritableMemory wmem = WritableMemory.allocate(bytes);
- shared = bldr.buildSharedFromSketch(recoveredShared, wmem);
- UpdateSketch local2 = bldr.buildLocal(shared);
+ shared = sl.bldr.buildSharedFromSketch(recoveredShared, wmem);
+ UpdateSketch local2 = sl.bldr.buildLocal(shared);
assertEquals(local2.getEstimate(), localEst);
assertEquals(local2.getLowerBound(2), localLB);
@@ -268,18 +258,16 @@ public class ConcurrentHeapQuickSelectSketchTest {
@Test
public void checkHQStoCompactForms() {
- int k = 512;
- lgK = 9;
+ int lgK = 9;
+ int k = 1 << lgK;
int u = 4*k;
boolean estimating = (u > k);
int maxBytes = (k << 4) + (Family.QUICKSELECT.getMinPreLongs() << 3);
- seed = DEFAULT_UPDATE_SEED;
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(null);
- UpdateSketch local = bldr.buildLocal(shared);
+ SharedLocal sl = new SharedLocal(lgK);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
assertEquals(local.getClass().getSimpleName(), "ConcurrentHeapThetaBuffer");
assertFalse(local.isDirect());
@@ -288,7 +276,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
for (int i=0; i<u; i++) {
local.update(i);
}
- waitForBgPropagationToComplete();
+ waitForBgPropagationToComplete(shared);
shared.rebuild(); //forces size back to k
@@ -296,9 +284,9 @@ public class ConcurrentHeapQuickSelectSketchTest {
double localEst = local.getEstimate();
double localLB = local.getLowerBound(2);
double localUB = local.getUpperBound(2);
- int localBytes = local.getCurrentBytes(); //size stored as UpdateSketch
- int localCompBytes = local.getCompactBytes(); //size stored as CompactSketch
- assertEquals(localBytes, maxBytes);
+ int sharedBytes = shared.getCurrentBytes();
+ int sharedCompBytes = shared.getCompactBytes();
+ assertEquals(sharedBytes, maxBytes);
assertEquals(local.isEstimationMode(), estimating);
CompactSketch comp1, comp2, comp3, comp4;
@@ -310,7 +298,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(comp1.getUpperBound(2), localUB);
assertEquals(comp1.isEmpty(), false);
assertEquals(comp1.isEstimationMode(), estimating);
- assertEquals(comp1.getCompactBytes(), localCompBytes);
+ assertEquals(comp1.getCompactBytes(), sharedCompBytes);
assertEquals(comp1.getClass().getSimpleName(), "HeapCompactSketch");
comp2 = shared.compact(true, null);
@@ -320,10 +308,10 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(comp2.getUpperBound(2), localUB);
assertEquals(comp2.isEmpty(), false);
assertEquals(comp2.isEstimationMode(), estimating);
- assertEquals(comp2.getCompactBytes(), localCompBytes);
+ assertEquals(comp2.getCompactBytes(), sharedCompBytes);
assertEquals(comp2.getClass().getSimpleName(), "HeapCompactSketch");
- byte[] memArr2 = new byte[localCompBytes];
+ byte[] memArr2 = new byte[sharedCompBytes];
WritableMemory mem2 = WritableMemory.wrap(memArr2); //allocate mem for compact form
comp3 = shared.compact(false, mem2); //load the mem2
@@ -333,7 +321,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(comp3.getUpperBound(2), localUB);
assertEquals(comp3.isEmpty(), false);
assertEquals(comp3.isEstimationMode(), estimating);
- assertEquals(comp3.getCompactBytes(), localCompBytes);
+ assertEquals(comp3.getCompactBytes(), sharedCompBytes);
assertEquals(comp3.getClass().getSimpleName(), "DirectCompactSketch");
mem2.clear();
@@ -344,19 +332,17 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(comp4.getUpperBound(2), localUB);
assertEquals(comp4.isEmpty(), false);
assertEquals(comp4.isEstimationMode(), estimating);
- assertEquals(comp4.getCompactBytes(), localCompBytes);
+ assertEquals(comp4.getCompactBytes(), sharedCompBytes);
assertEquals(comp4.getClass().getSimpleName(), "DirectCompactSketch");
comp4.toString(false, true, 0, false);
}
@Test
public void checkHQStoCompactEmptyForms() {
- lgK = 9;
- seed = DEFAULT_UPDATE_SEED;
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(null);
- UpdateSketch local = bldr.buildLocal(shared);
+ int lgK = 9;
+ SharedLocal sl = new SharedLocal(lgK);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
println("lgArr: "+ local.getLgArrLongs());
//empty
@@ -396,20 +382,18 @@ public class ConcurrentHeapQuickSelectSketchTest {
@Test
public void checkExactMode() {
- lgK = 12;
- int u = 4096;
- seed = DEFAULT_UPDATE_SEED;
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(null);
- UpdateSketch local = bldr.buildLocal(shared);
+ int lgK = 12;
+ int u = 1 << lgK;
+ SharedLocal sl = new SharedLocal(lgK);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
assertTrue(local.isEmpty());
for (int i = 0; i< u; i++) {
local.update(i);
}
- waitForBgPropagationToComplete();
+ waitForBgPropagationToComplete(shared);
assertEquals(local.getEstimate(), u, 0.0);
assertEquals(shared.getRetainedEntries(false), u);
@@ -417,13 +401,11 @@ public class ConcurrentHeapQuickSelectSketchTest {
@Test
public void checkEstMode() {
- int k = 4096;
- lgK = 12;
- seed = DEFAULT_UPDATE_SEED;
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(null);
- UpdateSketch local = bldr.buildLocal(shared);
+ int lgK = 12;
+ int k = 1 << lgK;
+ SharedLocal sl = new SharedLocal(lgK);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
assertTrue(local.isEmpty());
@@ -431,24 +413,23 @@ public class ConcurrentHeapQuickSelectSketchTest {
for (int i = 0; i< u; i++) {
local.update(i);
}
- waitForBgPropagationToComplete();
+ waitForBgPropagationToComplete(shared);
final int retained = shared.getRetainedEntries(false);
- //final int retained = ((UpdateSketch) shared).getRetainedEntries(false);
assertTrue(retained > k);
- // in general it might be exactly k, but in this case must be greater
+ // it could be exactly k, but in this case must be greater
}
@Test
public void checkErrorBounds() {
- int k = 512;
- lgK = 12;
-
- seed = DEFAULT_UPDATE_SEED;
- UpdateSketch local = buildSharedReturnLocalSketch();
+ int lgK = 9;
+ int k = 1 << lgK;
+ SharedLocal sl = new SharedLocal(lgK);
+ UpdateSketch local = sl.local;
+ UpdateSketch shared = sl.shared;
//Exact mode
- int limit = (int)ConcurrentSharedThetaSketch.computeExactLimit(k, 0);
- for (int i = 0; i < limit; i++ ) {
+ //int limit = (int)ConcurrentSharedThetaSketch.computeExactLimit(lim, 0); //? ask Eshcar
+ for (int i = 0; i < k; i++ ) {
local.update(i);
}
@@ -459,12 +440,12 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(est, lb, 0.0);
//Est mode
- int u = 10*k;
- for (int i = limit; i < u; i++ ) {
+ int u = 2 * k;
+ for (int i = k; i < u; i++ ) {
local.update(i);
local.update(i); //test duplicate rejection
}
- waitForBgPropagationToComplete();
+ waitForBgPropagationToComplete(shared);
est = local.getEstimate();
lb = local.getLowerBound(2);
ub = local.getUpperBound(2);
@@ -474,14 +455,12 @@ public class ConcurrentHeapQuickSelectSketchTest {
@Test
public void checkRebuild() {
- int k = 16;
- lgK = 4;
-
- seed = DEFAULT_UPDATE_SEED;
- final UpdateSketchBuilder bldr = configureBuilder();
+ int lgK = 4;
+ int k = 1 << lgK;
+ SharedLocal sl = new SharedLocal(lgK);
//must build shared first
- shared = bldr.buildShared(null);
- UpdateSketch local = bldr.buildLocal(shared);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
assertTrue(local.isEmpty());
int t = ((ConcurrentHeapThetaBuffer)local).getHashTableThreshold();
@@ -489,7 +468,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
for (int i = 0; i< t; i++) {
local.update(i);
}
- waitForBgPropagationToComplete();
+ waitForBgPropagationToComplete(shared);
assertFalse(local.isEmpty());
assertTrue(local.getEstimate() > 0.0);
@@ -503,55 +482,43 @@ public class ConcurrentHeapQuickSelectSketchTest {
assertEquals(shared.getRetainedEntries(true), k);
}
- @Test(expectedExceptions = SketchesStateException.class)
+ @Test
public void checkBuilder() {
- lgK = 4;
-
- seed = DEFAULT_UPDATE_SEED;
- final UpdateSketchBuilder bldr = configureBuilderWithNominal();
- assertEquals(bldr.getLocalLgNominalEntries(), lgK);
- assertEquals(bldr.getLgNominalEntries(), lgK);
- println(bldr.toString());
- bldr.buildLocal(shared);
- }
-
- @Test(expectedExceptions = SketchesArgumentException.class)
- public void checkBuilderSmallLgNominal() {
- lgK = 1;
- seed = DEFAULT_UPDATE_SEED;
- configureBuilder();
+ int lgK = 4;
+ SharedLocal sl = new SharedLocal(lgK);
+ assertEquals(sl.bldr.getLocalLgNominalEntries(), lgK);
+ assertEquals(sl.bldr.getLgNominalEntries(), lgK);
+ println(sl.bldr.toString());
}
+ @SuppressWarnings("unused")
@Test(expectedExceptions = SketchesArgumentException.class)
public void checkBuilderSmallNominal() {
- lgK = 2;
- seed = DEFAULT_UPDATE_SEED;
- configureBuilderWithNominal();
+ int lgK = 2; //too small
+ new SharedLocal(lgK);
}
@Test(expectedExceptions = SketchesArgumentException.class)
public void checkNegativeHashes() {
- lgK = 9;
- UpdateSketch qs = buildSharedReturnLocalSketch();
- qs.hashUpdate(-1L);
+ int lgK = 9;
+ SharedLocal sl = new SharedLocal(lgK);
+ UpdateSketch local = sl.local;
+ local.hashUpdate(-1L);
}
@Test
public void checkResetAndStartingSubMultiple() {
- int k = 512;
- lgK = 9;
-
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared();
- UpdateSketch local = bldr.buildLocal(shared);
- //ConcurrentHeapThetaBuffer sk1 = (ConcurrentHeapThetaBuffer)usk; //for internal checks
+ int lgK = 9;
+ int k = 1 << lgK;
+ SharedLocal sl = new SharedLocal(lgK);
+ UpdateSketch shared = sl.shared;
+ UpdateSketch local = sl.local;
assertTrue(local.isEmpty());
int u = 3*k;
for (int i = 0; i< u; i++) { local.update(i); }
- waitForBgPropagationToComplete();
+ waitForBgPropagationToComplete(shared);
assertFalse(local.isEmpty());
assertTrue(shared.getRetainedEntries(false) >= k);
@@ -567,12 +534,10 @@ public class ConcurrentHeapQuickSelectSketchTest {
@Test
public void checkDQStoCompactEmptyForms() {
- lgK = 9;
-
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(null);
- UpdateSketch local = bldr.buildLocal(shared);
+ int lgK = 9;
+ SharedLocal sl = new SharedLocal(lgK);
+ UpdateSketch local = sl.local;
+ UpdateSketch shared = sl.shared;
//empty
local.toString(false, true, 0, false); //exercise toString
@@ -582,7 +547,7 @@ public class ConcurrentHeapQuickSelectSketchTest {
double uskUB = local.getUpperBound(2);
assertFalse(local.isEstimationMode());
- int bytes = local.getCompactBytes(); //compact form
+ int bytes = local.getCompactBytes();
assertEquals(bytes, 8);
byte[] memArr2 = new byte[bytes];
WritableMemory mem2 = WritableMemory.wrap(memArr2);
@@ -608,75 +573,68 @@ public class ConcurrentHeapQuickSelectSketchTest {
@Test(expectedExceptions = SketchesArgumentException.class)
public void checkMinReqBytes() {
- int k = 16;
- lgK = 4;
- UpdateSketch local = buildSharedReturnLocalSketch();
- for (int i = 0; i < (4 * k); i++) { local.update(i); }
- waitForBgPropagationToComplete();
- byte[] byteArray = shared.toByteArray();
- byte[] badBytes = Arrays.copyOfRange(byteArray, 0, 24);
+ int lgK = 4;
+ int k = 1 << lgK;
+ SharedLocal sl = new SharedLocal(lgK);
+ for (int i = 0; i < (4 * k); i++) { sl.local.update(i); }
+ waitForBgPropagationToComplete(sl.shared);
+ byte[] byteArray = sl.shared.toByteArray();
+ byte[] badBytes = Arrays.copyOfRange(byteArray, 0, 24); //corrupt no. bytes
Memory mem = Memory.wrap(badBytes);
Sketch.heapify(mem);
}
@Test(expectedExceptions = SketchesArgumentException.class)
public void checkThetaAndLgArrLongs() {
- int k = 16;
- lgK = 4;
- UpdateSketch local = buildSharedReturnLocalSketch();
- for (int i = 0; i < k; i++) { local.update(i); }
- waitForBgPropagationToComplete();
- byte[] badArray = shared.toByteArray();
+ int lgK = 4;
+ int k = 1 << lgK;
+ SharedLocal sl = new SharedLocal(lgK);
+ for (int i = 0; i < k; i++) { sl.local.update(i); }
+ waitForBgPropagationToComplete(sl.shared);
+ byte[] badArray = sl.shared.toByteArray();
WritableMemory mem = WritableMemory.wrap(badArray);
- PreambleUtil.insertLgArrLongs(mem, 4);
- PreambleUtil.insertThetaLong(mem, Long.MAX_VALUE / 2);
+ PreambleUtil.insertLgArrLongs(mem, 4); //corrupt
+ PreambleUtil.insertThetaLong(mem, Long.MAX_VALUE / 2); //corrupt
Sketch.heapify(mem);
}
@Test
public void checkFamily() {
- UpdateSketch local = buildSharedReturnLocalSketch();
+ SharedLocal sl = new SharedLocal();
+ UpdateSketch local = sl.local;
assertEquals(local.getFamily(), Family.QUICKSELECT);
}
@Test
public void checkBackgroundPropagation() {
- int k = 16;
- lgK = 4;
+ int lgK = 4;
+ int k = 1 << lgK;
int u = 5*k;
- final UpdateSketchBuilder bldr = configureBuilderWithCache();
- //must build shared first
- shared = bldr.buildShared(null);
- UpdateSketch local = bldr.buildLocal(shared);
-
-
- assertTrue(local.isEmpty());
+ SharedLocal sl = new SharedLocal(lgK);
+ assertTrue(sl.local.isEmpty());
int i = 0;
- for (; i< k; i++) {
- local.update(i);
- }
- waitForBgPropagationToComplete();
- assertFalse(local.isEmpty());
- assertTrue(local.getEstimate() > 0.0);
- long theta1 = ((ConcurrentHeapQuickSelectSketch)shared).getVolatileTheta();
+ for (; i < k; i++) { sl.local.update(i); } //exact
+ waitForBgPropagationToComplete(sl.shared);
- for (; i< u; i++) {
- local.update(i);
- }
- waitForBgPropagationToComplete();
+ assertFalse(sl.local.isEmpty());
+ assertTrue(sl.local.getEstimate() > 0.0);
+ long theta1 = sl.sharedIf.getVolatileTheta();
- long theta2 = ((ConcurrentHeapQuickSelectSketch)shared).getVolatileTheta();
- int entries = shared.getRetainedEntries(false);
+ for (; i < u; i++) { sl.local.update(i); } //continue, make it estimating
+ waitForBgPropagationToComplete(sl.shared);
+
+ long theta2 = sl.sharedIf.getVolatileTheta();
+ int entries = sl.shared.getRetainedEntries(false);
assertTrue((entries > k) || (theta2 < theta1),
"entries= " + entries + " k= " + k + " theta1= " + theta1 + " theta2= " + theta2);
- shared.rebuild();
- assertEquals(shared.getRetainedEntries(false), k);
- assertEquals(shared.getRetainedEntries(true), k);
- local.rebuild();
- assertEquals(shared.getRetainedEntries(false), k);
- assertEquals(shared.getRetainedEntries(true), k);
+ sl.shared.rebuild();
+ assertEquals(sl.shared.getRetainedEntries(false), k);
+ assertEquals(sl.shared.getRetainedEntries(true), k);
+ sl.local.rebuild();
+ assertEquals(sl.shared.getRetainedEntries(false), k);
+ assertEquals(sl.shared.getRetainedEntries(true), k);
}
@Test
@@ -704,10 +662,8 @@ public class ConcurrentHeapQuickSelectSketchTest {
@Test(expectedExceptions = UnsupportedOperationException.class)
public void checkToByteArray() {
- UpdateSketchBuilder bldr = new UpdateSketchBuilder();
- UpdateSketch shared = bldr.buildShared();
- UpdateSketch local = bldr.buildLocal(shared);
- local.toByteArray();
+ SharedLocal sl = new SharedLocal();
+ sl.local.toByteArray();
}
@Test
@@ -715,11 +671,6 @@ public class ConcurrentHeapQuickSelectSketchTest {
println("PRINTING: "+this.getClass().getName());
}
- @AfterMethod
- public void clearShared() {
- shared = null;
- }
-
/**
* @param s value to print
*/
@@ -727,58 +678,71 @@ public class ConcurrentHeapQuickSelectSketchTest {
//System.out.println(s); //disable here
}
- private UpdateSketch buildSharedReturnLocalSketch() {
- final UpdateSketchBuilder bldr = configureBuilder();
- //must build shared first
- shared = bldr.buildShared(null);
- return bldr.buildLocal(shared);
- }
-
- //configures builder for both local and shared
- private UpdateSketchBuilder configureBuilder() {
+ static class SharedLocal {
+ static final long DefaultSeed = DEFAULT_UPDATE_SEED;
+ final UpdateSketch shared;
+ final ConcurrentSharedThetaSketch sharedIf;
+ final UpdateSketch local;
+ final int sharedLgK;
+ final int localLgK;
+ final long seed;
+ final WritableMemory wmem;
final UpdateSketchBuilder bldr = new UpdateSketchBuilder();
- bldr.setLogNominalEntries(lgK);
- bldr.setLocalLogNominalEntries(lgK);
- bldr.setSeed(seed);
- return bldr;
- }
- //configures builder for both local and shared
- private UpdateSketchBuilder configureBuilderNotOrdered() {
- final UpdateSketchBuilder bldr = new UpdateSketchBuilder();
- bldr.setLogNominalEntries(lgK);
- bldr.setLocalLogNominalEntries(4);
- bldr.setSeed(seed);
- bldr.setPropagateOrderedCompact(false);
- return bldr;
- }
+ SharedLocal() {
+ this(9, 9, DefaultSeed, false, true, 1);
+ }
- //configures builder for both local and shared
- private UpdateSketchBuilder configureBuilderWithNominal() {
- final UpdateSketchBuilder bldr = configureBuilder();
- int k = 1 << lgK;
- bldr.setLocalNominalEntries(k);
- bldr.setNominalEntries(k);
- assertTrue(bldr.getPropagateOrderedCompact());
- assertEquals(bldr.getSeed(), DEFAULT_UPDATE_SEED);
- return bldr;
- }
+ SharedLocal(int lgK) {
+ this(lgK, lgK, DefaultSeed, false, true, 1);
+ }
- //configures builder for both local and shared
- private UpdateSketchBuilder configureBuilderWithCache() {
- final UpdateSketchBuilder bldr = configureBuilder();
- return bldr;
+ SharedLocal(int sharedLgK, int localLgK) {
+ this(sharedLgK, localLgK, DefaultSeed, false, true, 1);
+ }
+
+ SharedLocal(int sharedLgK, int localLgK, long seed) {
+ this(sharedLgK, localLgK, seed, false, true, 1);
+ }
+
+ SharedLocal(int sharedLgK, int localLgK, boolean useMem) {
+ this(sharedLgK, localLgK, DefaultSeed, useMem, true, 1);
+ }
+
+ SharedLocal(int sharedLgK, int localLgK, boolean useMem, boolean ordered) {
+ this(sharedLgK, localLgK, DefaultSeed, useMem, ordered, 1);
+ }
+
+ SharedLocal(int sharedLgK, int localLgK, long seed, boolean useMem, boolean ordered, int memMult) {
+ this.sharedLgK = sharedLgK;
+ this.localLgK = localLgK;
+ this.seed = seed;
+ if (useMem) {
+ int bytes = (((4 << sharedLgK) * memMult) + (Family.QUICKSELECT.getMaxPreLongs())) << 3;
+ wmem = WritableMemory.allocate(bytes);
+ } else {
+ wmem = null;
+ }
+ bldr.setLogNominalEntries(sharedLgK);
+ bldr.setLocalLogNominalEntries(localLgK);
+ bldr.setPropagateOrderedCompact(ordered);
+ bldr.setSeed(this.seed);
+ shared = bldr.buildShared(wmem);
+ local = bldr.buildLocal(shared);
+ sharedIf = (ConcurrentSharedThetaSketch) shared;
+ }
}
- private void waitForBgPropagationToComplete() {
+ static void waitForBgPropagationToComplete(UpdateSketch shared) {
try {
Thread.sleep(10);
} catch (InterruptedException e) {
e.printStackTrace();
}
- ((ConcurrentHeapQuickSelectSketch)shared).awaitBgPropagationTermination();
+ ConcurrentSharedThetaSketch csts = (ConcurrentSharedThetaSketch)shared;
+ csts.awaitBgPropagationTermination();
ConcurrentPropagationService.resetExecutorService(Thread.currentThread().getId());
- ((ConcurrentHeapQuickSelectSketch)shared).initBgPropagationService();
+ csts.initBgPropagationService();
}
}
diff --git a/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java b/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java
index 0eaf1e4..26e7b4c 100644
--- a/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/DirectQuickSelectSketchTest.java
@@ -133,7 +133,7 @@ public class DirectQuickSelectSketchTest {
assertTrue(sk1.hasMemory());
assertFalse(sk1.isDirty());
assertTrue(sk1.hasMemory());
- assertEquals(sk1.getCurrentPreambleLongs(false), 3);
+ assertEquals(sk1.getCurrentPreambleLongs(), 3);
UpdateSketch sk2 = Sketches.heapifyUpdateSketch(mem);
assertEquals(sk2.getEstimate(), sk1est);
diff --git a/src/test/java/org/apache/datasketches/theta/DirectUnionTest.java b/src/test/java/org/apache/datasketches/theta/DirectUnionTest.java
index 0975b40..eb372a0 100644
--- a/src/test/java/org/apache/datasketches/theta/DirectUnionTest.java
+++ b/src/test/java/org/apache/datasketches/theta/DirectUnionTest.java
@@ -780,7 +780,7 @@ public class DirectUnionTest {
CompactSketch csk = union1.getResult();
assertTrue(csk.getTheta() < 0.2);
- assertEquals(csk.getRetainedEntries(), 16384);
+ assertEquals(csk.getRetainedEntries(true), 16384);
final double est = csk.getEstimate();
assertTrue(est > 98663.0);
assertTrue(est < 101530.0);
@@ -806,7 +806,7 @@ public class DirectUnionTest {
CompactSketch csk = union1.getResult();
assertTrue(csk.getTheta() < 0.2);
- assertEquals(csk.getRetainedEntries(), 16384);
+ assertEquals(csk.getRetainedEntries(true), 16384);
final double est = csk.getEstimate();
assertTrue(est > 98663.0);
assertTrue(est < 101530.0);
diff --git a/src/test/java/org/apache/datasketches/theta/SetOpsCornerCasesTest.java b/src/test/java/org/apache/datasketches/theta/SetOpsCornerCasesTest.java
index d29a84d..73180b0 100644
--- a/src/test/java/org/apache/datasketches/theta/SetOpsCornerCasesTest.java
+++ b/src/test/java/org/apache/datasketches/theta/SetOpsCornerCasesTest.java
@@ -329,15 +329,15 @@ public class SetOpsCornerCasesTest {
CompactSketch skHeap2 = generate(EST_HEAP, k);
CompactSketch csk;
csk = PairwiseSetOperations.union(skNull, skHeap1, k);
- Assert.assertEquals(csk.getRetainedEntries(), k);
+ Assert.assertEquals(csk.getRetainedEntries(true), k);
csk = PairwiseSetOperations.union(skEmpty, skHeap1, k);
- Assert.assertEquals(csk.getRetainedEntries(), k);
+ Assert.assertEquals(csk.getRetainedEntries(true), k);
csk = PairwiseSetOperations.union(skHeap1, skNull, k);
- Assert.assertEquals(csk.getRetainedEntries(), k);
+ Assert.assertEquals(csk.getRetainedEntries(true), k);
csk = PairwiseSetOperations.union(skHeap1, skEmpty, k);
- Assert.assertEquals(csk.getRetainedEntries(), k);
+ Assert.assertEquals(csk.getRetainedEntries(true), k);
csk = PairwiseSetOperations.union(skHeap1, skHeap2, k);
- Assert.assertEquals(csk.getRetainedEntries(), k);
+ Assert.assertEquals(csk.getRetainedEntries(true), k);
}
@@ -366,7 +366,7 @@ public class SetOpsCornerCasesTest {
csk = generate(State.EMPTY, k);
assertEquals(csk.isEmpty(), true);
assertEquals(csk.isEstimationMode(), false);
- assertEquals(csk.getRetainedEntries(), 0);
+ assertEquals(csk.getRetainedEntries(true), 0);
assertEquals(csk.getThetaLong(), Long.MAX_VALUE);
assertEquals(csk.isDirect(), false);
assertEquals(csk.hasMemory(), false);
@@ -375,7 +375,7 @@ public class SetOpsCornerCasesTest {
csk = generate(State.SINGLE, k);
assertEquals(csk.isEmpty(), false);
assertEquals(csk.isEstimationMode(), false);
- assertEquals(csk.getRetainedEntries(), 1);
+ assertEquals(csk.getRetainedEntries(true), 1);
assertEquals(csk.getThetaLong(), Long.MAX_VALUE);
assertEquals(csk.isDirect(), false);
assertEquals(csk.hasMemory(), false);
@@ -384,7 +384,7 @@ public class SetOpsCornerCasesTest {
csk = generate(State.EXACT, k);
assertEquals(csk.isEmpty(), false);
assertEquals(csk.isEstimationMode(), false);
- assertEquals(csk.getRetainedEntries(), k);
+ assertEquals(csk.getRetainedEntries(true), k);
assertEquals(csk.getThetaLong(), Long.MAX_VALUE);
assertEquals(csk.isDirect(), false);
assertEquals(csk.hasMemory(), false);
@@ -393,7 +393,7 @@ public class SetOpsCornerCasesTest {
csk = generate(State.EST_HEAP, k);
assertEquals(csk.isEmpty(), false);
assertEquals(csk.isEstimationMode(), true);
- assertEquals(csk.getRetainedEntries() > k, true);
+ assertEquals(csk.getRetainedEntries(true) > k, true);
assertEquals(csk.getThetaLong() < Long.MAX_VALUE, true);
assertEquals(csk.isDirect(), false);
assertEquals(csk.hasMemory(), false);
@@ -402,7 +402,7 @@ public class SetOpsCornerCasesTest {
csk = generate(State.THLT1_CNT0_FALSE, k);
assertEquals(csk.isEmpty(), false);
assertEquals(csk.isEstimationMode(), true);
- assertEquals(csk.getRetainedEntries(), 0);
+ assertEquals(csk.getRetainedEntries(true), 0);
assertEquals(csk.getThetaLong() < Long.MAX_VALUE, true);
assertEquals(csk.isDirect(), false);
assertEquals(csk.hasMemory(), false);
@@ -411,7 +411,7 @@ public class SetOpsCornerCasesTest {
csk = generate(State.THEQ1_CNT0_TRUE, k);
assertEquals(csk.isEmpty(), true);
assertEquals(csk.isEstimationMode(), false);
- assertEquals(csk.getRetainedEntries(), 0);
+ assertEquals(csk.getRetainedEntries(true), 0);
assertEquals(csk.getThetaLong() < Long.MAX_VALUE, false);
assertEquals(csk.isDirect(), false);
assertEquals(csk.hasMemory(), false);
@@ -420,7 +420,7 @@ public class SetOpsCornerCasesTest {
csk = generate(State.EST_MEMORY_UNORDERED, k);
assertEquals(csk.isEmpty(), false);
assertEquals(csk.isEstimationMode(), true);
- assertEquals(csk.getRetainedEntries() > k, true);
+ assertEquals(csk.getRetainedEntries(true) > k, true);
assertEquals(csk.getThetaLong() < Long.MAX_VALUE, true);
assertEquals(csk.isDirect(), false);
assertEquals(csk.hasMemory(), true);
@@ -467,13 +467,13 @@ public class SetOpsCornerCasesTest {
case THLT1_CNT0_FALSE : {
sk = Sketches.updateSketchBuilder().setP((float)0.5).setNominalEntries(k).build();
sk.update(7); //above theta
- assert(sk.getRetainedEntries() == 0);
+ assert(sk.getRetainedEntries(true) == 0);
csk = sk.compact(true, null); //compact as {Th < 1.0, 0, F}
break;
}
case THEQ1_CNT0_TRUE : {
sk = Sketches.updateSketchBuilder().setP((float)0.5).setNominalEntries(k).build();
- assert(sk.getRetainedEntries() == 0);
+ assert(sk.getRetainedEntries(true) == 0);
csk = sk.compact(true, null); //compact as {Th < 1.0, 0, T}
break;
}
diff --git a/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java b/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java
index d2fa4db..01226bf 100644
--- a/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/SingleItemSketchTest.java
@@ -167,7 +167,7 @@ public class SingleItemSketchTest {
public void checkRestricted() {
SingleItemSketch sis = SingleItemSketch.create(1);
assertNull(sis.getMemory());
- assertEquals(sis.getCurrentPreambleLongs(true), 1);
+ assertEquals(sis.getCompactPreambleLongs(), 1);
}
@Test
diff --git a/src/test/java/org/apache/datasketches/theta/SketchTest.java b/src/test/java/org/apache/datasketches/theta/SketchTest.java
index fa11112..1193e38 100644
--- a/src/test/java/org/apache/datasketches/theta/SketchTest.java
+++ b/src/test/java/org/apache/datasketches/theta/SketchTest.java
@@ -65,8 +65,8 @@ public class SketchTest {
int lowQSPreLongs = Family.QUICKSELECT.getMinPreLongs();
int lowCompPreLongs = Family.COMPACT.getMinPreLongs();
UpdateSketch sketch = UpdateSketch.builder().setNominalEntries(k).build(); // QS Sketch
- assertEquals(sketch.getCurrentPreambleLongs(false), lowQSPreLongs);
- assertEquals(sketch.getCurrentPreambleLongs(true), 1); //compact form
+ assertEquals(sketch.getCurrentPreambleLongs(), lowQSPreLongs);
+ assertEquals(sketch.getCompactPreambleLongs(), 1); //compact form
assertEquals(sketch.getCurrentDataLongs(), k*2);
assertEquals(sketch.getCurrentBytes(), (k*2*8) + (lowQSPreLongs << 3));
assertEquals(sketch.getCompactBytes(), lowCompPreLongs << 3);
@@ -84,8 +84,8 @@ public class SketchTest {
sketch.update(i);
}
- assertEquals(sketch.getCurrentPreambleLongs(false), lowQSPreLongs);
- assertEquals(sketch.getCurrentPreambleLongs(true), 2); //compact form
+ assertEquals(sketch.getCurrentPreambleLongs(), lowQSPreLongs);
+ assertEquals(sketch.getCompactPreambleLongs(), 2); //compact form
assertEquals(sketch.getCurrentDataLongs(), k*2);
assertEquals(sketch.getCurrentBytes(), (k*2*8) + (lowQSPreLongs << 3));
assertEquals(sketch.getCompactBytes(), (k*8) + (2*8)); //compact form //FAILS HERE
@@ -99,8 +99,8 @@ public class SketchTest {
}
int curCount = sketch.getRetainedEntries(true);
- assertEquals(sketch.getCurrentPreambleLongs(false), lowQSPreLongs);
- assertEquals(sketch.getCurrentPreambleLongs(true), 3); //compact form
+ assertEquals(sketch.getCurrentPreambleLongs(), lowQSPreLongs);
+ assertEquals(sketch.getCompactPreambleLongs(), 3); //compact form
assertEquals(sketch.getCurrentDataLongs(), k*2);
assertEquals(sketch.getCurrentBytes(), (k*2*8) + (lowQSPreLongs << 3));
assertEquals(sketch.getCompactBytes(), (curCount*8) + (3*8)); //compact form
diff --git a/src/test/java/org/apache/datasketches/theta/UnionImplTest.java b/src/test/java/org/apache/datasketches/theta/UnionImplTest.java
index c7d2b50..9ae9740 100644
--- a/src/test/java/org/apache/datasketches/theta/UnionImplTest.java
+++ b/src/test/java/org/apache/datasketches/theta/UnionImplTest.java
@@ -197,7 +197,7 @@ public class UnionImplTest {
for (int i = 0; i < k; i++) { sk.update(i); }
double est1 = sk.getEstimate();
- int bytes = Sketches.getMaxCompactSketchBytes(sk.getRetainedEntries());
+ int bytes = Sketches.getMaxCompactSketchBytes(sk.getRetainedEntries(true));
try (WritableDirectHandle h = WritableMemory.allocateDirect(bytes)) {
WritableMemory wmem = h.get();
CompactSketch csk = sk.compact(true, wmem); //ordered, direct
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[incubator-datasketches-java] 09/10: Close to final set of changes.
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leerho pushed a commit to branch Refactor_Theta_Tuple
in repository https://gitbox.apache.org/repos/asf/incubator-datasketches-java.git
commit e3f4ad2dee4d75356f587a4143634ac862c3493e
Author: Lee Rhodes <le...@users.noreply.github.com>
AuthorDate: Tue Jul 7 14:38:28 2020 -0700
Close to final set of changes.
---
.../org/apache/datasketches/HashOperations.java | 193 +++++++++++----------
.../theta/DirectQuickSelectSketch.java | 2 +-
.../theta/DirectQuickSelectSketchR.java | 18 +-
.../datasketches/theta/HeapQuickSelectSketch.java | 13 +-
.../datasketches/theta/HeapUpdateSketch.java | 12 +-
.../datasketches/theta/IntersectionImpl.java | 2 +-
.../java/org/apache/datasketches/theta/Sketch.java | 19 +-
.../apache/datasketches/theta/UpdateSketch.java | 39 ++---
.../datasketches/tuple/QuickSelectSketch.java | 39 +++--
.../DirectArrayOfDoublesQuickSelectSketch.java | 6 +-
.../apache/datasketches/HashOperationsTest.java | 23 +--
11 files changed, 194 insertions(+), 172 deletions(-)
diff --git a/src/main/java/org/apache/datasketches/HashOperations.java b/src/main/java/org/apache/datasketches/HashOperations.java
index 47bf16a..9067b79 100644
--- a/src/main/java/org/apache/datasketches/HashOperations.java
+++ b/src/main/java/org/apache/datasketches/HashOperations.java
@@ -33,7 +33,6 @@ import org.apache.datasketches.memory.WritableMemory;
*/
public final class HashOperations {
private static final int STRIDE_HASH_BITS = 7;
-
private static final int EMPTY = 0;
/**
@@ -43,47 +42,8 @@ public final class HashOperations {
private HashOperations() {}
- /**
- * Counts the cardinality of the first Log2 values of the given source array.
- * @param srcArr the given source array
- * @param lgArrLongs <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>
- * @param thetaLong <a href="{@docRoot}/resources/dictionary.html#thetaLong">See Theta Long</a>
- * @return the cardinality
- */
- public static int countPart(final long[] srcArr, final int lgArrLongs, final long thetaLong) {
- int cnt = 0;
- final int len = 1 << lgArrLongs;
- for (int i = len; i-- > 0;) {
- final long hash = srcArr[i];
- if (continueCondition(thetaLong, hash) ) {
- continue;
- }
- cnt++ ;
- }
- return cnt;
- }
-
- /**
- * Counts the cardinality of the given source array.
- * @param srcArr the given source array
- * @param thetaLong <a href="{@docRoot}/resources/dictionary.html#thetaLong">See Theta Long</a>
- * @return the cardinality
- */
- public static int count(final long[] srcArr, final long thetaLong) {
- int cnt = 0;
- final int len = srcArr.length;
- for (int i = len; i-- > 0;) {
- final long hash = srcArr[i];
- if (continueCondition(thetaLong, hash) ) {
- continue;
- }
- cnt++ ;
- }
- return cnt;
- }
-
- // make odd and independent of index assuming lgArrLongs lowest bits of the hash were used for
- // index
+ //Make odd and independent of index assuming lgArrLongs lowest bits of the hash were used for
+ // index. This results in a 8 bit value that is always odd.
private static int getStride(final long hash, final int lgArrLongs) {
return (2 * (int) ((hash >>> lgArrLongs) & STRIDE_MASK) ) + 1;
}
@@ -91,18 +51,18 @@ public final class HashOperations {
//ON-HEAP
/**
- * This is a classical Knuth-style Open Addressing, Double Hash search scheme for on-heap.
+ * This is a classical Knuth-style Open Addressing, Double Hash (OADH) search scheme for on-heap.
* Returns the index if found, -1 if not found.
*
- * @param hashTable The hash table to search. Must be a power of 2 in size.
- * @param lgArrLongs <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>.
- * lgArrLongs ≤ log2(hashTable.length).
- * @param hash A hash value to search for. Must not be zero.
+ * @param hashTable The hash table to search. Its size must be a power of 2.
+ * @param lgArrLongs The log_base2(hashTable.length).
+ * <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>.
+ * @param hash The hash value to search for. It must not be zero.
* @return Current probe index if found, -1 if not found.
*/
public static int hashSearch(final long[] hashTable, final int lgArrLongs, final long hash) {
if (hash == 0) {
- throw new SketchesArgumentException("Given hash cannot be zero: " + hash);
+ throw new SketchesArgumentException("Given hash must not be zero: " + hash);
}
final int arrayMask = (1 << lgArrLongs) - 1; // current Size -1
final int stride = getStride(hash, lgArrLongs);
@@ -123,16 +83,16 @@ public final class HashOperations {
}
/**
- * This is a classical Knuth-style Open Addressing, Double Hash insert scheme for on-heap.
+ * This is a classical Knuth-style Open Addressing, Double Hash (OADH) insert scheme for on-heap.
* This method assumes that the input hash is not a duplicate.
* Useful for rebuilding tables to avoid unnecessary comparisons.
- * Returns the index of insertion, which is always positive or zero. Throws an exception if the
- * table is full with no empty slot.
+ * Returns the index of insertion, which is always positive or zero.
+ * Throws an exception if the table has no empty slot.
*
- * @param hashTable the hash table to insert into.
- * @param lgArrLongs <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>.
- * lgArrLongs ≤ log2(hashTable.length).
- * @param hash value that must not be zero and will be inserted into the array into an empty slot.
+ * @param hashTable the hash table to insert into. Its size must be a power of 2.
+ * @param lgArrLongs The log_base2(hashTable.length).
+ * <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>.
+ * @param hash The hash value to be potentially inserted into an empty slot. It must not be zero.
* @return index of insertion. Always positive or zero.
*/
public static int hashInsertOnly(final long[] hashTable, final int lgArrLongs, final long hash) {
@@ -153,15 +113,15 @@ public final class HashOperations {
}
/**
- * This is a classical Knuth-style Open Addressing, Double Hash insert scheme for on-heap.
+ * This is a classical Knuth-style Open Addressing, Double Hash (OADH) insert scheme for on-heap.
* Returns index ≥ 0 if found (duplicate); < 0 if inserted, inserted at -(index + 1).
* Throws an exception if the value is not found and table has no empty slot.
*
- * @param hashTable the hash table to insert into.
- * @param lgArrLongs <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>.
- * lgArrLongs ≤ log2(hashTable.length).
- * @param hash hash value that must not be zero and if not a duplicate will be inserted into the
- * array into an empty slot
+ * @param hashTable The hash table to insert into. Its size must be a power of 2.
+ * @param lgArrLongs The log_base2(hashTable.length).
+ * <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>.
+ * @param hash The hash value to be potentially inserted into an empty slot only if it is not
+ * a duplicate of any other hash value in the table. It must not be zero.
* @return index ≥ 0 if found (duplicate); < 0 if inserted, inserted at -(index + 1).
*/
public static int hashSearchOrInsert(final long[] hashTable, final int lgArrLongs,
@@ -182,22 +142,24 @@ public final class HashOperations {
}
curProbe = (curProbe + stride) & arrayMask;
} while (curProbe != loopIndex);
- throw new SketchesArgumentException("Key not found and no empty slots!");
+ throw new SketchesArgumentException("Hash not found and no empty slots!");
}
/**
- * Inserts the given long array into the given hash table array of the target size,
- * removes any negative input values, ignores duplicates and counts the values inserted.
+ * Inserts the given long array into the given OADH hashTable of the target size,
+ * ignores duplicates and counts the values inserted.
+ * The hash values must not be negative, zero values and values ≥ thetaLong are ignored.
* The given hash table may have values, but they must have been inserted by this method or one
- * of the other OADH insert methods in this class and they may not be dirty.
+ * of the other OADH insert methods in this class.
* This method performs additional checks against potentially invalid hash values or theta values.
* Returns the count of values actually inserted.
*
* @param srcArr the source hash array to be potentially inserted
- * @param hashTable The correctly sized target hash table that must be a power of two.
- * @param lgArrLongs <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>.
- * lgArrLongs ≤ log2(hashTable.length).
- * @param thetaLong must greater than zero
+ * @param hashTable The hash table to insert into. Its size must be a power of 2.
+ * @param lgArrLongs The log_base2(hashTable.length).
+ * <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>.
+ * @param thetaLong The theta value that all input hash values are compared against.
+ * It must greater than zero.
* <a href="{@docRoot}/resources/dictionary.html#thetaLong">See Theta Long</a>
* @return the count of values actually inserted
*/
@@ -219,21 +181,25 @@ public final class HashOperations {
return count;
}
- //OFF-HEAP (these are kept for backward compatibility)
+ //With Memory or WritableMemory
/**
- * This is a classical Knuth-style Open Addressing, Double Hash search scheme for off-heap.
+ * This is a classical Knuth-style Open Addressing, Double Hash (OADH) search scheme for Memory.
* Returns the index if found, -1 if not found.
*
- * @param mem The Memory hash table to search.
- * @param lgArrLongs <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>.
- * lgArrLongs ≤ log2(hashTable.length).
- * @param hash A hash value to search for. Must not be zero.
- * @param memOffsetBytes offset in the memory where the hash array starts
- * @return index if found, -1 if not found.
+ * @param mem The <i>Memory</i> containing the hash table to search.
+ * The hash table portion must be a power of 2 in size.
+ * @param lgArrLongs The log_base2(hashTable.length).
+ * <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>.
+ * @param hash The hash value to search for. Must not be zero.
+ * @param memOffsetBytes offset in the memory where the hashTable starts
+ * @return Current probe index if found, -1 if not found.
*/
- public static int hashSearch(final Memory mem, final int lgArrLongs, final long hash,
+ public static int hashSearchMemory(final Memory mem, final int lgArrLongs, final long hash,
final int memOffsetBytes) {
+ if (hash == 0) {
+ throw new SketchesArgumentException("Given hash must not be zero: " + hash);
+ }
final int arrayMask = (1 << lgArrLongs) - 1;
final int stride = getStride(hash, lgArrLongs);
int curProbe = (int) (hash & arrayMask);
@@ -249,21 +215,21 @@ public final class HashOperations {
}
/**
- * This is a classical Knuth-style Open Addressing, Double Hash insert scheme, but inserts
- * values directly into a Memory.
+ * This is a classical Knuth-style Open Addressing, Double Hash (OADH) insert scheme for Memory.
* This method assumes that the input hash is not a duplicate.
* Useful for rebuilding tables to avoid unnecessary comparisons.
* Returns the index of insertion, which is always positive or zero.
* Throws an exception if table has no empty slot.
*
- * @param wmem The writable memory
- * @param lgArrLongs <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>.
- * lgArrLongs ≤ log2(hashTable.length).
+ * @param wmem The <i>WritableMemory</i> that contains the hashTable to insert into.
+ * The size of the hashTable portion must be a power of 2.
+ * @param lgArrLongs The log_base2(hashTable.length.
+ * <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>.
* @param hash value that must not be zero and will be inserted into the array into an empty slot.
- * @param memOffsetBytes offset in the memory where the hash array starts
+ * @param memOffsetBytes offset in the <i>WritableMemory</i> where the hashTable starts
* @return index of insertion. Always positive or zero.
*/
- public static int fastHashInsertOnly(final WritableMemory wmem, final int lgArrLongs,
+ public static int hashInsertOnlyMemory(final WritableMemory wmem, final int lgArrLongs,
final long hash, final int memOffsetBytes) {
final int arrayMask = (1 << lgArrLongs) - 1; // current Size -1
final int stride = getStride(hash, lgArrLongs);
@@ -282,23 +248,21 @@ public final class HashOperations {
throw new SketchesArgumentException("No empty slot in table!");
}
- //FAST OFF-HEAP
-
/**
* This is a classical Knuth-style Open Addressing, Double Hash insert scheme, but inserts
* values directly into a Memory.
* Returns index ≥ 0 if found (duplicate); < 0 if inserted, inserted at -(index + 1).
* Throws an exception if the value is not found and table has no empty slot.
*
- * @param wmem the WritableMemory
- * @param lgArrLongs <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>.
- * lgArrLongs ≤ log2(hashTable.length).
- * @param hash A hash value that must not be zero and if not a duplicate will be inserted into the
- * array into an empty slot.
- * @param memOffsetBytes offset in the memory where the hash array starts
+ * @param wmem The <i>WritableMemory</i> that contains the hashTable to insert into.
+ * @param lgArrLongs The log_base2(hashTable.length).
+ * <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>.
+ * @param hash The hash value to be potentially inserted into an empty slot only if it is not
+ * a duplicate of any other hash value in the table. It must not be zero.
+ * @param memOffsetBytes offset in the <i>WritableMemory</i> where the hash array starts
* @return index ≥ 0 if found (duplicate); < 0 if inserted, inserted at -(index + 1).
*/
- public static int fastHashSearchOrInsert(final WritableMemory wmem, final int lgArrLongs,
+ public static int hashSearchOrInsertMemory(final WritableMemory wmem, final int lgArrLongs,
final long hash, final int memOffsetBytes) {
final int arrayMask = (1 << lgArrLongs) - 1; // current Size -1
final int stride = getStride(hash, lgArrLongs);
@@ -318,6 +282,8 @@ public final class HashOperations {
throw new SketchesArgumentException("Key not found and no empty slot in table!");
}
+ //Other related methods
+
/**
* @param thetaLong must be greater than zero otherwise throws an exception.
* <a href="{@docRoot}/resources/dictionary.html#thetaLong">See Theta Long</a>
@@ -378,4 +344,43 @@ public final class HashOperations {
return hashTable;
}
+ /**
+ * Counts the cardinality of the first Log2 values of the given source array.
+ * @param srcArr the given source array
+ * @param lgArrLongs <a href="{@docRoot}/resources/dictionary.html#lgArrLongs">See lgArrLongs</a>
+ * @param thetaLong <a href="{@docRoot}/resources/dictionary.html#thetaLong">See Theta Long</a>
+ * @return the cardinality
+ */
+ public static int countPart(final long[] srcArr, final int lgArrLongs, final long thetaLong) {
+ int cnt = 0;
+ final int len = 1 << lgArrLongs;
+ for (int i = len; i-- > 0;) {
+ final long hash = srcArr[i];
+ if (continueCondition(thetaLong, hash) ) {
+ continue;
+ }
+ cnt++ ;
+ }
+ return cnt;
+ }
+
+ /**
+ * Counts the cardinality of the given source array.
+ * @param srcArr the given source array
+ * @param thetaLong <a href="{@docRoot}/resources/dictionary.html#thetaLong">See Theta Long</a>
+ * @return the cardinality
+ */
+ public static int count(final long[] srcArr, final long thetaLong) {
+ int cnt = 0;
+ final int len = srcArr.length;
+ for (int i = len; i-- > 0;) {
+ final long hash = srcArr[i];
+ if (continueCondition(thetaLong, hash) ) {
+ continue;
+ }
+ cnt++ ;
+ }
+ return cnt;
+ }
+
}
diff --git a/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketch.java b/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketch.java
index dc34bc5..df24c11 100644
--- a/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketch.java
@@ -260,7 +260,7 @@ class DirectQuickSelectSketch extends DirectQuickSelectSketchR {
//The duplicate test
final int index =
- HashOperations.fastHashSearchOrInsert(wmem_, lgArrLongs, hash, preambleLongs << 3);
+ HashOperations.hashSearchOrInsertMemory(wmem_, lgArrLongs, hash, preambleLongs << 3);
if (index >= 0) {
return RejectedDuplicate; //Duplicate, not inserted
}
diff --git a/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java b/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java
index d5290b1..d79b71b 100644
--- a/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java
+++ b/src/main/java/org/apache/datasketches/theta/DirectQuickSelectSketchR.java
@@ -188,11 +188,21 @@ class DirectQuickSelectSketchR extends UpdateSketch {
}
@Override
+ float getP() {
+ return wmem_.getFloat(P_FLOAT);
+ }
+
+ @Override
public ResizeFactor getResizeFactor() {
return ResizeFactor.getRF(getLgRF());
}
@Override
+ long getSeed() {
+ return seed_;
+ }
+
+ @Override
public UpdateSketch rebuild() {
throw new SketchesReadOnlyException();
}
@@ -229,15 +239,7 @@ class DirectQuickSelectSketchR extends UpdateSketch {
return wmem_;
}
- @Override
- float getP() {
- return wmem_.getFloat(P_FLOAT);
- }
- @Override
- long getSeed() {
- return seed_;
- }
@Override
short getSeedHash() {
diff --git a/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java b/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java
index b3dcf6b..14197aa 100644
--- a/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/HeapQuickSelectSketch.java
@@ -280,14 +280,15 @@ class HeapQuickSelectSketch extends HeapUpdateSketch {
return numEntries > hashTableThreshold_;
}
- //Must resize. Changes lgArrLongs_ and cache_. theta and count don't change.
+ //Must resize. Changes lgArrLongs_, cache_, hashTableThreshold;
+ // theta and count don't change.
// Used by hashUpdate()
private final void resizeCache() {
final ResizeFactor rf = getResizeFactor();
- final int lgTgtLongs = lgNomLongs_ + 1;
- final int lgDeltaLongs = lgTgtLongs - lgArrLongs_;
+ final int lgMaxArrLongs = lgNomLongs_ + 1;
+ final int lgDeltaLongs = lgMaxArrLongs - lgArrLongs_;
final int lgResizeFactor = max(min(rf.lg(), lgDeltaLongs), 1); //rf_.lg() could be 0
- lgArrLongs_ += lgResizeFactor; // new tgt size
+ lgArrLongs_ += lgResizeFactor; // new arr size
final long[] tgtArr = new long[1 << lgArrLongs_];
final int newCount = HashOperations.hashArrayInsert(cache_, tgtArr, lgArrLongs_, thetaLong_);
@@ -301,9 +302,9 @@ class HeapQuickSelectSketch extends HeapUpdateSketch {
//array stays the same size. Changes theta and thus count
private final void quickSelectAndRebuild() {
- final int arrLongs = 1 << lgArrLongs_;
+ final int arrLongs = 1 << lgArrLongs_; // generally 2 * k,
- final int pivot = (1 << lgNomLongs_) + 1; // pivot for QS
+ final int pivot = (1 << lgNomLongs_) + 1; // pivot for QS = k + 1
thetaLong_ = selectExcludingZeros(cache_, curCount_, pivot); //messes up the cache_
diff --git a/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java b/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java
index d62e88d..2bc2915 100644
--- a/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java
@@ -85,15 +85,13 @@ abstract class HeapUpdateSketch extends UpdateSketch {
}
@Override
- public ResizeFactor getResizeFactor() {
- return rf_;
+ float getP() {
+ return p_;
}
- //restricted methods
-
@Override
- float getP() {
- return p_;
+ public ResizeFactor getResizeFactor() {
+ return rf_;
}
@Override
@@ -101,6 +99,8 @@ abstract class HeapUpdateSketch extends UpdateSketch {
return seed_;
}
+ //restricted methods
+
@Override
short getSeedHash() {
return Util.computeSeedHash(getSeed());
diff --git a/src/main/java/org/apache/datasketches/theta/IntersectionImpl.java b/src/main/java/org/apache/datasketches/theta/IntersectionImpl.java
index d5478e4..611130d 100644
--- a/src/main/java/org/apache/datasketches/theta/IntersectionImpl.java
+++ b/src/main/java/org/apache/datasketches/theta/IntersectionImpl.java
@@ -359,7 +359,7 @@ final class IntersectionImpl extends IntersectionImplR {
for (int i = 0; i < arrLongsIn; i++ ) {
final long hashIn = arr[i];
if (HashOperations.continueCondition(thetaLong, hashIn)) { continue; }
- HashOperations.fastHashInsertOnly(mem_, lgArrLongs, hashIn, preBytes);
+ HashOperations.hashInsertOnlyMemory(mem_, lgArrLongs, hashIn, preBytes);
tmpCnt++;
}
} else { //On Heap. Assumes HT exists and is large enough
diff --git a/src/main/java/org/apache/datasketches/theta/Sketch.java b/src/main/java/org/apache/datasketches/theta/Sketch.java
index d65c5cb..0075b82 100644
--- a/src/main/java/org/apache/datasketches/theta/Sketch.java
+++ b/src/main/java/org/apache/datasketches/theta/Sketch.java
@@ -280,14 +280,6 @@ public abstract class Sketch {
public abstract Family getFamily();
/**
- * Returns a HashIterator that can be used to iterate over the retained hash values of the
- * Theta sketch.
- * @return a HashIterator that can be used to iterate over the retained hash values of the
- * Theta sketch.
- */
- public abstract HashIterator iterator();
-
- /**
* Gets the approximate lower error bound given the specified number of Standard Deviations.
* This will return getEstimate() if isEmpty() is true.
*
@@ -437,6 +429,14 @@ public abstract class Sketch {
}
/**
+ * Returns a HashIterator that can be used to iterate over the retained hash values of the
+ * Theta sketch.
+ * @return a HashIterator that can be used to iterate over the retained hash values of the
+ * Theta sketch.
+ */
+ public abstract HashIterator iterator();
+
+ /**
* Serialize this sketch to a byte array form.
* @return byte array of this sketch
*/
@@ -568,7 +568,7 @@ public abstract class Sketch {
/**
* Gets the internal cache array. For on-heap sketches this will return a reference to the actual
- * cache array. For off-heap sketches this returns a copy.
+ * cache array. For Memory-based sketches this returns a copy.
* @return the internal cache array.
*/
abstract long[] getCache();
@@ -633,7 +633,6 @@ public abstract class Sketch {
}
}
-
static final double estimate(final long thetaLong, final int curCount) {
return curCount * (LONG_MAX_VALUE_AS_DOUBLE / thetaLong);
}
diff --git a/src/main/java/org/apache/datasketches/theta/UpdateSketch.java b/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
index c28c983..740b03e 100644
--- a/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
+++ b/src/main/java/org/apache/datasketches/theta/UpdateSketch.java
@@ -167,7 +167,6 @@ public abstract class UpdateSketch extends Sketch {
/**
* Returns a new builder
- *
* @return a new builder
*/
public static final UpdateSketchBuilder builder() {
@@ -175,6 +174,25 @@ public abstract class UpdateSketch extends Sketch {
}
/**
+ * Returns the configured ResizeFactor
+ * @return the configured ResizeFactor
+ */
+ public abstract ResizeFactor getResizeFactor();
+
+ /**
+ * Gets the configured sampling probability, <i>p</i>.
+ * <a href="{@docRoot}/resources/dictionary.html#p">See Sampling Probability, <i>p</i></a>
+ * @return the sampling probability, <i>p</i>
+ */
+ abstract float getP();
+
+ /**
+ * Gets the configured seed
+ * @return the configured seed
+ */
+ abstract long getSeed();
+
+ /**
* Resets this sketch back to a virgin empty state.
*/
public abstract void reset();
@@ -187,12 +205,6 @@ public abstract class UpdateSketch extends Sketch {
public abstract UpdateSketch rebuild();
/**
- * Returns the configured ResizeFactor
- * @return the configured ResizeFactor
- */
- public abstract ResizeFactor getResizeFactor();
-
- /**
* Present this sketch with a long.
*
* @param datum The given long datum.
@@ -330,19 +342,6 @@ public abstract class UpdateSketch extends Sketch {
public abstract int getLgNomLongs();
/**
- * Gets the configured sampling probability, <i>p</i>.
- * <a href="{@docRoot}/resources/dictionary.html#p">See Sampling Probability, <i>p</i></a>
- * @return the sampling probability, <i>p</i>
- */
- abstract float getP();
-
- /**
- * Gets the configured seed
- * @return the configured seed
- */
- abstract long getSeed();
-
- /**
* Returns true if the internal cache contains "dirty" values that are greater than or equal
* to thetaLong.
* @return true if the internal cache is dirty.
diff --git a/src/main/java/org/apache/datasketches/tuple/QuickSelectSketch.java b/src/main/java/org/apache/datasketches/tuple/QuickSelectSketch.java
index 26abe61..e6fb943 100644
--- a/src/main/java/org/apache/datasketches/tuple/QuickSelectSketch.java
+++ b/src/main/java/org/apache/datasketches/tuple/QuickSelectSketch.java
@@ -41,7 +41,6 @@ import org.apache.datasketches.memory.Memory;
* @param <S> type of Summary
*/
class QuickSelectSketch<S extends Summary> extends Sketch<S> {
- //private static final byte serialVersionWithSummaryFactoryUID = 1;
private static final byte serialVersionUID = 2;
private enum Flags { IS_BIG_ENDIAN, IS_IN_SAMPLING_MODE, IS_EMPTY, HAS_ENTRIES, IS_THETA_INCLUDED }
@@ -63,7 +62,9 @@ class QuickSelectSketch<S extends Summary> extends Sketch<S> {
* given value.
* @param summaryFactory An instance of a SummaryFactory.
*/
- QuickSelectSketch(final int nomEntries, final SummaryFactory<S> summaryFactory) {
+ QuickSelectSketch(
+ final int nomEntries,
+ final SummaryFactory<S> summaryFactory) {
this(nomEntries, DEFAULT_LG_RESIZE_FACTOR, summaryFactory);
}
@@ -80,7 +81,9 @@ class QuickSelectSketch<S extends Summary> extends Sketch<S> {
* </pre>
* @param summaryFactory An instance of a SummaryFactory.
*/
- QuickSelectSketch(final int nomEntries, final int lgResizeFactor,
+ QuickSelectSketch(
+ final int nomEntries,
+ final int lgResizeFactor,
final SummaryFactory<S> summaryFactory) {
this(nomEntries, lgResizeFactor, 1f, summaryFactory);
}
@@ -100,7 +103,10 @@ class QuickSelectSketch<S extends Summary> extends Sketch<S> {
* @param samplingProbability the given sampling probability
* @param summaryFactory An instance of a SummaryFactory.
*/
- QuickSelectSketch(final int nomEntries, final int lgResizeFactor, final float samplingProbability,
+ QuickSelectSketch(
+ final int nomEntries,
+ final int lgResizeFactor,
+ final float samplingProbability,
final SummaryFactory<S> summaryFactory) {
this(
nomEntries,
@@ -111,8 +117,12 @@ class QuickSelectSketch<S extends Summary> extends Sketch<S> {
);
}
- QuickSelectSketch(final int nomEntries, final int lgResizeFactor, final float samplingProbability,
- final SummaryFactory<S> summaryFactory, final int startingSize) {
+ QuickSelectSketch(
+ final int nomEntries,
+ final int lgResizeFactor,
+ final float samplingProbability,
+ final SummaryFactory<S> summaryFactory,
+ final int startingSize) {
nomEntries_ = ceilingPowerOf2(nomEntries);
lgResizeFactor_ = lgResizeFactor;
samplingProbability_ = samplingProbability;
@@ -130,7 +140,9 @@ class QuickSelectSketch<S extends Summary> extends Sketch<S> {
* @param deserializer the SummaryDeserializer
* @param summaryFactory the SummaryFactory
*/
- QuickSelectSketch(final Memory mem, final SummaryDeserializer<S> deserializer,
+ QuickSelectSketch(
+ final Memory mem,
+ final SummaryDeserializer<S> deserializer,
final SummaryFactory<S> summaryFactory) {
summaryFactory_ = summaryFactory;
int offset = 0;
@@ -202,7 +214,6 @@ class QuickSelectSketch<S extends Summary> extends Sketch<S> {
return summaryTable_;
}
-
/**
* Get configured nominal number of entries
* @return nominal number of entries
@@ -249,7 +260,7 @@ class QuickSelectSketch<S extends Summary> extends Sketch<S> {
public void trim() {
if (count_ > nomEntries_) {
updateTheta();
- rebuild(hashTable_.length);
+ resize(hashTable_.length);
}
}
@@ -427,13 +438,13 @@ class QuickSelectSketch<S extends Summary> extends Sketch<S> {
updateTheta();
rebuild();
} else {
- rebuild(hashTable_.length * (1 << lgResizeFactor_));
+ resize(hashTable_.length * (1 << lgResizeFactor_));
}
return true;
}
void rebuild() {
- rebuild(hashTable_.length);
+ resize(hashTable_.length);
}
void insert(final long hash, final S summary) {
@@ -446,6 +457,10 @@ class QuickSelectSketch<S extends Summary> extends Sketch<S> {
private void updateTheta() {
final long[] hashArr = new long[count_];
int i = 0;
+ //Because of the association of the hashTable with the summaryTable we cannot destroy the
+ // hashTable structure. So we must copy. May as well compact at the same time.
+ // Might consider a whole table clone and use the selectExcludingZeros method instead.
+ // Not sure if there would be any speed advantage.
for (int j = 0; j < hashTable_.length; j++) {
if (summaryTable_[j] != null) {
hashArr[i++] = hashTable_[j];
@@ -455,7 +470,7 @@ class QuickSelectSketch<S extends Summary> extends Sketch<S> {
}
@SuppressWarnings({"unchecked"})
- private void rebuild(final int newSize) {
+ private void resize(final int newSize) {
final long[] oldHashTable = hashTable_;
final S[] oldSummaryTable = summaryTable_;
final Class<S> summaryType = (Class<S>) summaryTable_.getClass().getComponentType();
diff --git a/src/main/java/org/apache/datasketches/tuple/arrayofdoubles/DirectArrayOfDoublesQuickSelectSketch.java b/src/main/java/org/apache/datasketches/tuple/arrayofdoubles/DirectArrayOfDoublesQuickSelectSketch.java
index c47dd1a..5c4e4b4 100644
--- a/src/main/java/org/apache/datasketches/tuple/arrayofdoubles/DirectArrayOfDoublesQuickSelectSketch.java
+++ b/src/main/java/org/apache/datasketches/tuple/arrayofdoubles/DirectArrayOfDoublesQuickSelectSketch.java
@@ -294,17 +294,17 @@ class DirectArrayOfDoublesQuickSelectSketch extends ArrayOfDoublesQuickSelectSke
@Override
protected int insertKey(final long key) {
- return HashOperations.fastHashInsertOnly(mem_, lgCurrentCapacity_, key, ENTRIES_START);
+ return HashOperations.hashInsertOnlyMemory(mem_, lgCurrentCapacity_, key, ENTRIES_START);
}
@Override
protected int findOrInsertKey(final long key) {
- return HashOperations.fastHashSearchOrInsert(mem_, lgCurrentCapacity_, key, ENTRIES_START);
+ return HashOperations.hashSearchOrInsertMemory(mem_, lgCurrentCapacity_, key, ENTRIES_START);
}
@Override
protected double[] find(final long key) {
- final int index = HashOperations.hashSearch(mem_, lgCurrentCapacity_, key, ENTRIES_START);
+ final int index = HashOperations.hashSearchMemory(mem_, lgCurrentCapacity_, key, ENTRIES_START);
if (index == -1) { return null; }
final double[] array = new double[numValues_];
mem_.getDoubleArray(valuesOffset_ + ((long) SIZE_OF_VALUE_BYTES * numValues_ * index),
diff --git a/src/test/java/org/apache/datasketches/HashOperationsTest.java b/src/test/java/org/apache/datasketches/HashOperationsTest.java
index 00c34fa..f7495aa 100644
--- a/src/test/java/org/apache/datasketches/HashOperationsTest.java
+++ b/src/test/java/org/apache/datasketches/HashOperationsTest.java
@@ -22,11 +22,12 @@ package org.apache.datasketches;
import static org.apache.datasketches.HashOperations.checkHashCorruption;
import static org.apache.datasketches.HashOperations.checkThetaCorruption;
import static org.apache.datasketches.HashOperations.continueCondition;
-import static org.apache.datasketches.HashOperations.fastHashInsertOnly;
-import static org.apache.datasketches.HashOperations.fastHashSearchOrInsert;
+import static org.apache.datasketches.HashOperations.hashInsertOnlyMemory;
+import static org.apache.datasketches.HashOperations.hashSearchOrInsertMemory;
import static org.apache.datasketches.HashOperations.hashArrayInsert;
import static org.apache.datasketches.HashOperations.hashInsertOnly;
import static org.apache.datasketches.HashOperations.hashSearch;
+import static org.apache.datasketches.HashOperations.hashSearchMemory;
import static org.apache.datasketches.HashOperations.hashSearchOrInsert;
import static org.apache.datasketches.hash.MurmurHash3.hash;
import static org.testng.Assert.assertEquals;
@@ -106,7 +107,7 @@ public class HashOperationsTest {
public void testHashInsertOnlyMemoryNoStride() {
final long[] table = new long[32];
final WritableMemory mem = WritableMemory.wrap(table);
- final int index = fastHashInsertOnly(mem, 5, 1, 0);
+ final int index = hashInsertOnlyMemory(mem, 5, 1, 0);
assertEquals(index, 1);
assertEquals(table[1], 1L);
}
@@ -116,7 +117,7 @@ public class HashOperationsTest {
final long[] table = new long[32];
table[1] = 1;
final WritableMemory mem = WritableMemory.wrap(table);
- final int index = fastHashInsertOnly(mem, 5, 1, 0);
+ final int index = hashInsertOnlyMemory(mem, 5, 1, 0);
assertEquals(index, 2);
assertEquals(table[2], 1L);
}
@@ -152,22 +153,22 @@ public class HashOperationsTest {
final long[] table = new long[32];
final WritableMemory mem = WritableMemory.wrap(table);
for (int i = 1; i <= 32; ++i) {
- fastHashInsertOnly(mem, 5, i, 0);
+ hashInsertOnlyMemory(mem, 5, i, 0);
}
// table full; search returns not found, others throw exception
- final int retVal = hashSearch(mem, 5, 33, 0);
+ final int retVal = hashSearchMemory(mem, 5, 33, 0);
assertEquals(retVal, -1);
try {
- fastHashInsertOnly(mem, 5, 33, 0);
+ hashInsertOnlyMemory(mem, 5, 33, 0);
fail();
} catch (final SketchesArgumentException e) {
// expected
}
try {
- fastHashSearchOrInsert(mem, 5, 33, 0);
+ hashSearchOrInsertMemory(mem, 5, 33, 0);
fail();
} catch (final SketchesArgumentException e) {
// expected
@@ -180,19 +181,19 @@ public class HashOperationsTest {
final WritableMemory wmem = WritableMemory.wrap(table);
for (int i = 1; i <= 32; ++i) {
- fastHashInsertOnly(wmem, 5, i, 0);
+ hashInsertOnlyMemory(wmem, 5, i, 0);
}
// table full; throws exception
try {
- fastHashInsertOnly(wmem, 5, 33, 0);
+ hashInsertOnlyMemory(wmem, 5, 33, 0);
fail();
} catch (final SketchesArgumentException e) {
// expected
}
try {
- fastHashSearchOrInsert(wmem, 5, 33, 0);
+ hashSearchOrInsertMemory(wmem, 5, 33, 0);
fail();
} catch (final SketchesArgumentException e) {
// expected
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[incubator-datasketches-java] 07/10: Merge branch 'master' into
Refactor_Theta_Tuple
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leerho pushed a commit to branch Refactor_Theta_Tuple
in repository https://gitbox.apache.org/repos/asf/incubator-datasketches-java.git
commit 46746a8133669c77f9db52e84ba3ce1f2a4e379c
Merge: 8736b03 eead07c
Author: Lee Rhodes <le...@users.noreply.github.com>
AuthorDate: Thu Jul 2 18:09:17 2020 -0700
Merge branch 'master' into Refactor_Theta_Tuple
src/main/java/org/apache/datasketches/theta/Sketch.java | 1 +
.../java/org/apache/datasketches/tuple/QuickSelectSketch.java | 2 +-
.../tuple/arrayofdoubles/ArrayOfDoublesQuickSelectSketch.java | 4 ++--
.../apache/datasketches/tuple/adouble/AdoubleAnotBTest.java | 2 +-
.../datasketches/tuple/aninteger/IntegerSketchTest.java | 11 ++++++-----
5 files changed, 11 insertions(+), 9 deletions(-)
diff --cc src/main/java/org/apache/datasketches/theta/Sketch.java
index 251abc8,7e7be3a..d65c5cb
--- a/src/main/java/org/apache/datasketches/theta/Sketch.java
+++ b/src/main/java/org/apache/datasketches/theta/Sketch.java
@@@ -633,8 -589,61 +633,9 @@@ public abstract class Sketch
}
}
- /*
- * The truth table for empty, curCount and theta when compacting is as follows:
- * <pre>
- * Num Theta CurCount Empty State Comments
- * 0 1.0 0 T OK The Normal Empty State
- * 1 1.0 0 F Internal This can result from an intersection of two exact, disjoint sets,
- * or AnotB of two exact, identical sets. There is no probability
- * distribution, so change to empty. Return {Th = 1.0, 0, T}.
- * This is handled in SetOperation.createCompactSketch().
- * 2 1.0 !0 T Error Empty=T and curCount !0 should never co-exist.
- * This is checked in all compacting operations.
- * 3 1.0 !0 F OK This corresponds to a sketch in exact mode
- * 4 <1.0 0 T Internal This can be an initial UpdateSketch state if p < 1.0,
- * so change theta to 1.0. Return {Th = 1.0, 0, T}.
- * This is handled in UpdateSketch.compact() and toByteArray().
- * 5 <1.0 0 F OK This can result from set operations
- * 6 <1.0 !0 T Error Empty=T and curCount !0 should never co-exist.
- * This is checked in all compacting operations.
- * 7 <1.0 !0 F OK This corresponds to a sketch in estimation mode
- * </pre>
- */
-
- /**
- * This checks for the illegal condition where curCount > 0 and the state of
- * empty = true. This check can be used anywhere a sketch is returned or a sketch is created
- * from complete arguments.
- * @param empty the given empty state
- * @param curCount the given current count
- */ //This handles #2 and #6 above
- static final void checkIllegalCurCountAndEmpty(final boolean empty, final int curCount) {
- if (empty && (curCount != 0)) { //this handles #2 and #6 above
- throw new SketchesStateException("Illegal State: Empty=true and Current Count != 0.");
- }
- }
-
- /**
- * This corrects a temporary anomalous condition where compact() is called on an UpdateSketch
- * that was initialized with p < 1.0 and update() was never called. In this case Theta < 1.0,
- * curCount = 0, and empty = true. The correction is to change Theta to 1.0, which makes the
- * returning sketch empty. This should only be used in the compaction or serialization of an
- * UpdateSketch.
- * @param empty the given empty state
- * @param curCount the given curCount
- * @param thetaLong the given thetaLong
- * @return thetaLong
- */ //This handles #4 above
- static final long correctThetaOnCompact(final boolean empty, final int curCount,
- final long thetaLong) {
- return (empty && (curCount == 0) && (thetaLong < Long.MAX_VALUE)) ? Long.MAX_VALUE : thetaLong;
- }
-
-
+
static final double estimate(final long thetaLong, final int curCount) {
- return curCount * (MAX_THETA_LONG_AS_DOUBLE / thetaLong);
+ return curCount * (LONG_MAX_VALUE_AS_DOUBLE / thetaLong);
}
static final double lowerBound(final int curCount, final long thetaLong, final int numStdDev,
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org