You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/08/14 18:23:42 UTC
[GitHub] [druid] suneet-s opened a new pull request #10285: Add note about aggregations on floats
suneet-s opened a new pull request #10285:
URL: https://github.com/apache/druid/pull/10285
Floating point math is known to be unstable. Due to the way aggregators work
across segments it's possible for the same query operating on the same data to
produce slightly different results.
The same problem exists with any aggregators that are not commutative since
the merge order across segments is not guaranteed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #10285: Add note about aggregations on floats
Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #10285:
URL: https://github.com/apache/druid/pull/10285#discussion_r471665827
##########
File path: docs/development/extensions-core/stats.md
##########
@@ -44,6 +44,13 @@ This algorithm was proven to be numerically stable by J.L. Barlow in
"Error analysis of a pairwise summation algorithm to compute sample variance"
Numer. Math, 58 (1991) pp. 583--590
+> As with all [aggregators](../../querying/sql.md#aggregation-functions), the order of operations across segments is
+> non-deterministic. This means that if this aggregator operates with an input type of "float" or "double", the result
+> of the aggregation will not be precisely the same across multiple runs of the query.
Review comment:
```suggestion
> of the aggregation may not be precisely the same across multiple runs of the query.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #10285: Add note about aggregations on floats
Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #10285:
URL: https://github.com/apache/druid/pull/10285#discussion_r471665956
##########
File path: docs/querying/sql.md
##########
@@ -232,6 +232,13 @@ possible for two aggregators in the same SQL query to have different filters.
Only the COUNT aggregation can accept DISTINCT.
+> The order of aggregation operations across segments is not deterministic. This means that non-commutative aggregation
+> functions can produce inconsistent results across the same query.
+>
+> Functions that operates on an input type of "float" or "double" may also see these differences in aggregation
Review comment:
```suggestion
> Functions that operate on an input type of "float" or "double" may also see these differences in aggregation
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm commented on a change in pull request #10285: Add note about aggregations on floats
Posted by GitBox <gi...@apache.org>.
gianm commented on a change in pull request #10285:
URL: https://github.com/apache/druid/pull/10285#discussion_r470809516
##########
File path: docs/development/extensions-core/stats.md
##########
@@ -44,6 +44,13 @@ This algorithm was proven to be numerically stable by J.L. Barlow in
"Error analysis of a pairwise summation algorithm to compute sample variance"
Numer. Math, 58 (1991) pp. 583--590
+> As with all [aggregators](../../querying/sql.md#aggregation-functions), the order of operations across segments is
+> non-deterministic. This means that if this aggregator is used with an input type of "float", the result of the
+> aggregation will not be precisely the same across multiple runs of the query.
+>
+> To produce consistent results, either cast the input type to a "double" or round the variance to
Review comment:
Double and float are both floating-point, so should behave the same way in this regard.
##########
File path: docs/querying/sql.md
##########
@@ -232,6 +232,10 @@ possible for two aggregators in the same SQL query to have different filters.
Only the COUNT aggregation can accept DISTINCT.
+> The order of aggregation operations across segments is not deterministic. This means that non-commutative aggregation
+> functions can produce inconsistent results across the same query. Any functions that operate on an input type of "float"
+> may also see these differences in aggregation results across multiple query runs.
Review comment:
Similar comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s merged pull request #10285: Add note about aggregations on floats
Posted by GitBox <gi...@apache.org>.
suneet-s merged pull request #10285:
URL: https://github.com/apache/druid/pull/10285
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org