You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/10/20 20:19:53 UTC

[GitHub] [druid] gianm opened a new pull request, #13247: Always return sketches from DS_HLL, DS_THETA, DS_QUANTILES_SKETCH.

gianm opened a new pull request, #13247:
URL: https://github.com/apache/druid/pull/13247

   These aggregation functions are documented as creating sketches. However, they are planned into native aggregators that include finalization logic to convert the sketch to a number of some sort. This creates an inconsistency: the functions sometimes return sketches, and sometimes return numbers, depending on where they lie in the native query plan.
   
   This patch changes these SQL aggregators to _never_ finalize, by using the "shouldFinalize" feature of the native aggregators. It already existed for theta sketches. This patch adds the feature for hll and quantiles sketches.
   
   As to impact, Druid finalizes aggregators in two cases:
   
   - When they appear in the outer level of a query (not a subquery).
   - When they are used as input to an expression or finalizing-field-access post-aggregator (not any other kind of post-aggregator).
   
   With this patch, the functions will no longer be finalized in these cases.
   
   The second item is not likely to matter much. The SQL functions all declare return type OTHER, which would be usable as an input to any other function that makes sense and that would be planned into an expression.
   
   So, the main effect of this patch is the first item. To provide backwards compatibility with anyone that was depending on the old behavior, the patch adds a "sqlFinalizeOuterSketches" query context parameter that restores the old behavior.
   
   Other changes:
   
   1) Move various argument-checking logic from runtime to planning time in
      DoublesSketchListArgBaseOperatorConversion, by adding an OperandTypeChecker.
   
   2) Add various JsonIgnores to the sketches to simplify their JSON representations.
   
   3) Allow chaining of ExpressionPostAggregators and other PostAggregators
      in the SQL layer.
   
   4) Avoid unnecessary FieldAccessPostAggregator wrapping in the SQL layer,
      now that expressions can operate on complex inputs.
   
   5) Adjust return type to thetaSketch (instead of OTHER) in
      ThetaSketchSetBaseOperatorConversion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] lgtm-com[bot] commented on pull request #13247: Always return sketches from DS_HLL, DS_THETA, DS_QUANTILES_SKETCH.

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13247:
URL: https://github.com/apache/druid/pull/13247#issuecomment-1299045254

   This pull request **introduces 1 alert** when merging 64e2eb098ee5b40a7e3d29132fca84c0d5c47ecd into 176934e849d50a2360f50db7771c1775110308f3 - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-7b9ff7463eb4be67ceb377b753ce320168c2f615)
   
   **new alerts:**
   
   * 1 for Inconsistent equals and hashCode


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] lgtm-com[bot] commented on pull request #13247: Always return sketches from DS_HLL, DS_THETA, DS_QUANTILES_SKETCH.

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13247:
URL: https://github.com/apache/druid/pull/13247#issuecomment-1286510635

   This pull request **introduces 1 alert** when merging 5de1c215ed150e95fbe70fb47f4a77f5c60bc995 into 86e6e61e884230e7a82e726f5f5c33602b242caf - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-f3329d08a2ed7d8eed76496f302714dac5ab3496)
   
   **new alerts:**
   
   * 1 for Inconsistent equals and hashCode


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] lgtm-com[bot] commented on pull request #13247: Always return sketches from DS_HLL, DS_THETA, DS_QUANTILES_SKETCH.

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13247:
URL: https://github.com/apache/druid/pull/13247#issuecomment-1301557434

   This pull request **introduces 1 alert** when merging 7cad8383dbfa60263adfdda6b8424559d77c9533 into 176934e849d50a2360f50db7771c1775110308f3 - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-a34ee9e37f187451365432084ce039dedff5aef7)
   
   **new alerts:**
   
   * 1 for Inconsistent equals and hashCode


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] lgtm-com[bot] commented on pull request #13247: Always return sketches from DS_HLL, DS_THETA, DS_QUANTILES_SKETCH.

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13247:
URL: https://github.com/apache/druid/pull/13247#issuecomment-1289343433

   This pull request **introduces 1 alert** when merging dac16627e2d457927ca3fcc9810c04cf8d16605f into d98c808d3f087645d8dcfd516e77b0058bdfd389 - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-9704534824ec01bcad21799ddc31eb2068925437)
   
   **new alerts:**
   
   * 1 for Inconsistent equals and hashCode


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] lgtm-com[bot] commented on pull request #13247: Always return sketches from DS_HLL, DS_THETA, DS_QUANTILES_SKETCH.

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13247:
URL: https://github.com/apache/druid/pull/13247#issuecomment-1290137568

   This pull request **introduces 1 alert** when merging 07af49b9c31cb9304b1d88bfdf91d68144740459 into 1e39bc65ccfb8e149330d7e873c274251ceed84a - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-10304175d364d8854e9fda506d3bcd80f9a26aee)
   
   **new alerts:**
   
   * 1 for Inconsistent equals and hashCode


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] gianm commented on pull request #13247: Always return sketches from DS_HLL, DS_THETA, DS_QUANTILES_SKETCH.

Posted by GitBox <gi...@apache.org>.
gianm commented on PR #13247:
URL: https://github.com/apache/druid/pull/13247#issuecomment-1286113734

   Marked as incompatible due to the behavioral change. There is the `sqlFinalizeOuterSketches` that people can use to restore the old behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] lgtm-com[bot] commented on pull request #13247: Always return sketches from DS_HLL, DS_THETA, DS_QUANTILES_SKETCH.

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13247:
URL: https://github.com/apache/druid/pull/13247#issuecomment-1294851689

   This pull request **introduces 1 alert** when merging dcbe0a3928db7459b0fdccf4598d042929d6dc9f into 32020247d12dd5776b83342848dd4a54cb339a9b - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-fe936e5d8cbb11ce5e95b8caa5a2bc4656d6aef8)
   
   **new alerts:**
   
   * 1 for Inconsistent equals and hashCode


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] gianm merged pull request #13247: Always return sketches from DS_HLL, DS_THETA, DS_QUANTILES_SKETCH.

Posted by GitBox <gi...@apache.org>.
gianm merged PR #13247:
URL: https://github.com/apache/druid/pull/13247


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org