You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datasketches.apache.org by GitBox <gi...@apache.org> on 2021/02/14 03:58:42 UTC
[GitHub] [datasketches-java] prateekiiest opened a new pull request #350: Add a method in item sketch to retrieve active keys
prateekiiest opened a new pull request #350:
URL: https://github.com/apache/datasketches-java/pull/350
This is a small addition to the list of current methods supported under ItemSketch.
In many applications, it's required to know at each stage what are the current active Items residing in the hashmap. Instead of indirectly calculating based on the current map size and all, we can get info directly regarding a list of current active keys residing in the hashmap.
Also using this we can get to know which key got removed from consecutive timestamps T1 and T2 or which new one got added.
sketch.getWhichKeysAreActive()_T1 - sketch.getWhichKeysAreActive()_T2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-java] jmalkin commented on pull request #350: Add a method in item sketch to retrieve active keys
Posted by GitBox <gi...@apache.org>.
jmalkin commented on pull request #350:
URL: https://github.com/apache/datasketches-java/pull/350#issuecomment-778813722
Can you explain more about why it's important to know non-frequent items at any given point in time? This will in general return a bunch of noise that would be filtered out using even the less-aggressive NO_FALSE_NEGATIVES flag. It does not provide a way to distinguish actual frequent items from a few items that happened to be added once but have low enough counts as to be meaningless.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-java] leerho commented on pull request #350: Add a method in item sketch to retrieve active keys
Posted by GitBox <gi...@apache.org>.
leerho commented on pull request #350:
URL: https://github.com/apache/datasketches-java/pull/350#issuecomment-780734003
I have opened a [discussion](https://lists.apache.org/thread.html/ra72ffa4d0729a4743f3ed742c5923dee072078ead9135084809a5b60%40%3Cdev.datasketches.apache.org%3E) on our dev@datasketches.apache.org list. Please join us there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-java] leerho closed pull request #350: Add a method in item sketch to retrieve active keys
Posted by GitBox <gi...@apache.org>.
leerho closed pull request #350:
URL: https://github.com/apache/datasketches-java/pull/350
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-java] prateekiiest commented on pull request #350: Add a method in item sketch to retrieve active keys
Posted by GitBox <gi...@apache.org>.
prateekiiest commented on pull request #350:
URL: https://github.com/apache/datasketches-java/pull/350#issuecomment-780002118
Hi @jmalkin, apologies for the late reply.
The obvious use case is that at any given time point, I can get an accurate instance of what are the current probes residing in the hash data structure, which I feel can not be tried out via any other current api calls/combination of them.
You are right that at any point the activeKeys might not be relevant to the entire stream as a whole considering a heavy hitter application, where initial such activeKeys will be counted as noise.
- An interesting utility that I am also exploring in my work is to capture these fluctuating keys, which occur intermittently due to lower than group median freq - thus classifying as noise. As compared to CMS sketches which stores instead every possible incoming item frequency (thus including this noise as well), it can hint towards by how much fraction is the Frequent Item sketch more accurate than CMS (might not be in all cases) since CMS contains the noise factor for applications like heavy hitters.
- Also in applications like anomaly detection, where any item from the stream might be anomalous, then for each item, we would like to calculate some custom anomaly score which in turn is determined by the individual item frequency over time. In such scenarios, it might be required to track each individual item and how it appears in the sketch over time.
These are some ideas that are coming to my mind and also during my work I am requiring use cases as such, although examples like these might be rare, and one might alternatively get to know if an item exists in the sketch if getEstimate(item) != nan.
Look forward to your thoughts / suggestions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-java] jmalkin commented on pull request #350: Add a method in item sketch to retrieve active keys
Posted by GitBox <gi...@apache.org>.
jmalkin commented on pull request #350:
URL: https://github.com/apache/datasketches-java/pull/350#issuecomment-779389436
The appropriate way to compare sketch contents over time would be to use `.getFrequentItems(ErrorType.NO_FALSE_NEGATIVES)` -- you'll certainly get some false positives, but you'll get anything that at least _might_ rise above the noise.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-java] leerho commented on pull request #350: Add a method in item sketch to retrieve active keys
Posted by GitBox <gi...@apache.org>.
leerho commented on pull request #350:
URL: https://github.com/apache/datasketches-java/pull/350#issuecomment-791722156
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-java] leerho commented on pull request #350: Add a method in item sketch to retrieve active keys
Posted by GitBox <gi...@apache.org>.
leerho commented on pull request #350:
URL: https://github.com/apache/datasketches-java/pull/350#issuecomment-780068133
@prateekiiest
Prateek,I am curious as to why you only want the array of active items and not their associated frequencies at a point in time. The current freq count will give you a lot of information about the potential stability of the item in the sketch. Also having access to the current ub and lb will enable you to infer the same conclusions the sketch makes about whether an item's lb exceeds the noise floor or just the ub, and thus it's likelihood of being a frequent item. I would think you would want what we already define as a Row, just not filtered.
We try to discourage users from trying to infer meaning to retained items that the sketch considers to be noise. That is why the Rows returned by the getFrequentItems(...) methods are filtered by an ErrorType and never include the noise. However, you apparently are performing more extensive analysis, even on the noisy items, that is beyond the scope of the design of the sketch.
If we are to do this, Instead of enabling just getActiveKeys(), which is essentially a hack to address your application, perhaps we should extend the ErrorType to include "ALL_ITEMS", so that the Rows returned by getFrequentItems(..) would contain everything. This would be a more general solution that may be useful to other applications in addition to yours.
I would like to open this issue more broadly to our community to get some other opinions.
Cheers,
Lee.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org