[GitHub] [druid] leerho edited a comment on issue #9736: HLLSketchMerge aggregator failing for some metrics after upgrade to v0.18

[GitBox <gi...@apache.org>]

leerho edited a comment on issue #9736:
URL: https://github.com/apache/druid/issues/9736#issuecomment-617480110


   Thanks for the stack trace.  It is very helpful. I think I may have a clue what may be happening.
   
   The specific exception thrown by the HLL union operator is: 
   
   > org.apache.datasketches.SketchesArgumentException: Incoming sketch is corrupted, Rebuild_CurMin_Num_KxQ flag is set.
   
   This occurs in only one place, which is the union.update(HllSketch input) method.  It is detecting that a special flag is set in the input sketch that is only ever set during a union operation.   Thus the incoming sketch had to be an image of a union operator and not that of a streaming updatable sketch nor a streaming compact sketch.  
   
   This flag is used to detect that the internal state of the union data structure is not finalized.  It is only finalized when you call 
   
   > HllSketch out = union.getResult(TgtHllType type);
   
   Using a serialized union operator as input to a union merge operation is not legal and it never has been.   The [example code](https://datasketches.apache.org/docs/HLL/HllJavaExample.html) also illustrates the use of union.getResult(type), and that example code has been there for several years.
   
   It is only with version 1.2.0 that this special case is properly detected. 
   
   There still might be a problem in the HLL code, but I would appreciate it if you could check your usage code and see if my hunch is correct.
   
   Lee.
   
   
   
   
   
   
   
   
   
   
   
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org