[datasketches-hive] branch master updated: Added documentation as requested by issue #54.

[le...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

leerho pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/datasketches-hive.git


The following commit(s) were added to refs/heads/master by this push:
     new 0a0cbb7  Added documentation as requested by issue #54.
0a0cbb7 is described below

commit 0a0cbb7a811f445c759bd56ea68c2a6e43034116
Author: Lee Rhodes <le...@users.noreply.github.com>
AuthorDate: Wed Feb 2 13:54:52 2022 -0800

    Added documentation as requested by issue #54.
---
 src/main/java/org/apache/datasketches/hive/hll/SketchState.java | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/src/main/java/org/apache/datasketches/hive/hll/SketchState.java b/src/main/java/org/apache/datasketches/hive/hll/SketchState.java
index 8cf58d6..808e658 100644
--- a/src/main/java/org/apache/datasketches/hive/hll/SketchState.java
+++ b/src/main/java/org/apache/datasketches/hive/hll/SketchState.java
@@ -62,7 +62,13 @@ class SketchState extends State {
         this.sketch_.update(PrimitiveObjectInspectorUtils.getLong(data, objectInspector));
         return;
       case STRING:
-        // conversion to char[] avoids costly UTF-8 encoding
+        // This gets the java String and hashes the underlying UTF-16 array. It was an early
+        //attempt at optimization, which unfortunately is different from all other String
+        //hashing in the library which assume UTF-8. This can't be changed due to all the
+        //history users have created using this current method. Users that don't want the
+        //string conversion here can either convert their strings to the underlying UTF-16
+        //and use case CHAR, or if their strings are already UTF-8, grab the underlying
+        //byte array and use the BINARY method.
         this.sketch_.update(PrimitiveObjectInspectorUtils.getString(data, objectInspector)
             .toCharArray());
         return;

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org