You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@datasketches.apache.org by "Fengzdadi (via GitHub)" <gi...@apache.org> on 2026/01/30 07:04:20 UTC

[I] VarOpt heapify: add validation for totalRWeight in full preamble (align with C++) (datasketches-java)

Fengzdadi opened a new issue, #721:
URL: https://github.com/apache/datasketches-java/issues/721

   ## Description
   In Java VarOptItemsSketch heapify, full‑preamble deserialization currently only checks that `rCount > 0`, but does not validate the `totalRWeight` field. C++ (`var_opt_sketch_impl.hpp`) rejects NaN or non‑positive `total_wt_r`, and Go was updated to follow C++ behavior. This creates a cross‑language inconsistency and allows corrupted or malicious bytes to silently produce NaN/negative tau and invalid estimates.
   
   ## Current Java code
   **File:** `src/main/java/org/apache/datasketches/sampling/VarOptItemsSketch.java`  
   **Method:** heapify (around where `numPreLongs == Family.VAROPT.getMaxPreLongs()`)
   
   Current logic:
   ```java
   double totalRWeight = 0.0;
   if (numPreLongs == Family.VAROPT.getMaxPreLongs()) {
     if (rCount > 0) {
       totalRWeight = extractTotalRWeight(srcSeg);
     } else {
       throw new SketchesArgumentException(
           "Possible Corruption: " + Family.VAROPT.getMaxPreLongs()
           + " preLongs but no items in R region");
     }
   }
   ```
   
   ## Behavior in other languages
   **C++** (`sampling/include/var_opt_sketch_impl.hpp`):
   - Full preamble → if `r == 0` or `total_wt_r` is NaN or <= 0, throw  
     ```
     "Possible corruption: deserializing in full mode but r = 0 or invalid R weight.
      Found r = ..., R region weight = ..."
     ```
   
   **Go** (`sampling/varopt_items_sketch.go`):
   - Full preamble → reject NaN or <= 0 totalRWeight (aligned with C++)
   
   ## Proposed change
   Add validation after `extractTotalRWeight` in Java:
   ```java
   double totalRWeight = 0.0;
   if (numPreLongs == Family.VAROPT.getMaxPreLongs()) {
     if (rCount > 0) {
       totalRWeight = extractTotalRWeight(srcSeg);
       if (Double.isNaN(totalRWeight) || totalRWeight <= 0.0) {
         throw new SketchesArgumentException(
             "Possible Corruption: deserializing in full mode but r = 0 or invalid R weight. "
             + "Found r = " + rCount + ", R region weight = " + totalRWeight);
       }
     } else {
       throw new SketchesArgumentException(
           "Possible Corruption: " + Family.VAROPT.getMaxPreLongs()
           + " preLongs but no items in R region");
     }
   }
   ```
   
   ## Why this matters
   - Prevents NaN/negative tau from silently propagating  
   - Aligns Java behavior with C++  
   - Only rejects invalid/corrupted bytes
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org


Re: [I] VarOpt heapify: add validation for totalRWeight in full preamble (align with C++) (datasketches-java)

Posted by "Fengzdadi (via GitHub)" <gi...@apache.org>.
Fengzdadi commented on issue #721:
URL: https://github.com/apache/datasketches-java/issues/721#issuecomment-3865309932

   Tks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org


Re: [I] VarOpt heapify: add validation for totalRWeight in full preamble (align with C++) (datasketches-java)

Posted by "leerho (via GitHub)" <gi...@apache.org>.
leerho commented on issue #721:
URL: https://github.com/apache/datasketches-java/issues/721#issuecomment-3862518175

   I'm ok with this.  Go ahead with the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org


Re: [I] VarOpt heapify: add validation for totalRWeight in full preamble (align with C++) (datasketches-java)

Posted by "Fengzdadi (via GitHub)" <gi...@apache.org>.
Fengzdadi closed issue #721: VarOpt heapify: add validation for totalRWeight in full preamble (align with C++)
URL: https://github.com/apache/datasketches-java/issues/721


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org