You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2016/09/16 13:58:15 UTC
[GitHub] incubator-metron pull request #257: METRON-426: Stellar does not support sci...
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/257
METRON-426: Stellar does not support scientific notation as a literal
Stellar does not support scientific notation for expressing literal doubles. It should.
This is exercised in the unit tests for `StellarStatisticsTest` and `StellarTest` directly
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron statistics_test_nondeterminism
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/257.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #257
----
commit 03420b8e309deb7262d122e3ed4a7972d1fe58b6
Author: cstella <ce...@gmail.com>
Date: 2016-09-16T13:56:18Z
METRON-426: Stellar does not support scientific notation as a literal
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #257: METRON-426: Stellar does not support scientific...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/257
The issue around the `StellarScientificTest` change is that merging t-digests is not precisely the same as adding the same data to one t-digest. Whether this is a bug in the t-digest code or whether it's an artifact of the theory, I don't know. Because this test is using low amounts of data, it's prone to a bit of jitter it seems and the compression level specified before isn't sufficient to ensure that the epsilon chosen (1e-2) is sufficient for repeated runs. I tightened up the compression level to 150 (which is still quite sensible) to make sure we can support at least 1e-2 accuracy for normally distributed data across 100 runs of 1000 elements.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #257: METRON-426: Stellar does not support sci...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/257#discussion_r79245698
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/math/stats/OnlineStatisticsProvider.java ---
@@ -43,7 +43,7 @@
* 100 is a sensible default and the number of centroids retained (to construct the sketch)
* is usually a smallish (usually < 10) multiple of the compression.
*/
- public static final int COMPRESSION = 100;
+ public static final int COMPRESSION = 150;
--- End diff --
yep
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #257: METRON-426: Stellar does not support scientific...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/257
+1 Less ego-driven. Ha.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #257: METRON-426: Stellar does not support sci...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/257
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #257: METRON-426: Stellar does not support sci...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/257#discussion_r79698302
--- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java ---
@@ -173,6 +173,22 @@ public void testMapConstant() {
}
@Test
+ public void testArithmetic() {
+ {
+ String query = "1 + 2";
+ Assert.assertEquals(3, ((Number)run(query, ImmutableMap.of("casey", "casey"))).doubleValue(), 1e-3);
+ }
+ {
+ String query = "1.2 + 2";
+ Assert.assertEquals(3.2, ((Number)run(query, ImmutableMap.of("casey", "casey"))).doubleValue(), 1e-3);
+ }
+ {
+ String query = "1.2e-3 + 2";
+ Assert.assertEquals(1.2e-3 + 2, ((Number)run(query, ImmutableMap.of("casey", "casey"))).doubleValue(), 1e-3);
+ }
--- End diff --
The `ImmutableMap.of("casey","casey")`(besides promoting your global brand ;) ) do nothing for these tests, right? It'd be more clear, if we removed those. Everything else looks good. I'm a +1 if we can take care of this small issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #257: METRON-426: Stellar does not support scientific...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/257
@justinleet I think everything you mentioned has been addressed. Are you good with this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #257: METRON-426: Stellar does not support sci...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/257#discussion_r79702619
--- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java ---
@@ -173,6 +173,22 @@ public void testMapConstant() {
}
@Test
+ public void testArithmetic() {
+ {
+ String query = "1 + 2";
+ Assert.assertEquals(3, ((Number)run(query, ImmutableMap.of("casey", "casey"))).doubleValue(), 1e-3);
+ }
+ {
+ String query = "1.2 + 2";
+ Assert.assertEquals(3.2, ((Number)run(query, ImmutableMap.of("casey", "casey"))).doubleValue(), 1e-3);
+ }
+ {
+ String query = "1.2e-3 + 2";
+ Assert.assertEquals(1.2e-3 + 2, ((Number)run(query, ImmutableMap.of("casey", "casey"))).doubleValue(), 1e-3);
+ }
--- End diff --
You got it. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #257: METRON-426: Stellar does not support sci...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/257#discussion_r79204587
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/math/stats/OnlineStatisticsProvider.java ---
@@ -43,7 +43,7 @@
* 100 is a sensible default and the number of centroids retained (to construct the sketch)
* is usually a smallish (usually < 10) multiple of the compression.
*/
- public static final int COMPRESSION = 100;
+ public static final int COMPRESSION = 150;
--- End diff --
Could you update the comment to reflect the compression default change?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---