[I] [STORM-7] storm.trident.operation.Aggregator: include group information in init() method (storm)

["jira-importer (via GitHub)" <gi...@apache.org>]

jira-importer opened a new issue, #3849:
URL: https://github.com/apache/storm/issues/3849

   <p>Reported by @lorenzfischer</p>
   
   <p>To be able to share resources between different groups in a grouped aggregator, it would be helpful to have information about the group available in the init() method of the aggregator interface.</p>
   
   <p>The concrete use case is the following:</p>
   
   <p>For our project we need to count the number of unique values in a field of a grouped stream. We have hundreds of millions of unique values and millions of grouped values. For this reason, we're currently deploying the HyperLogLog class that has generously been made available by the people at Clearspring >(<a href="https://github.com/clearspring/stream-lib" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/clearspring/stream-lib</a>). Naturally, we end up with millions of counter objects.</p>
   
   <p>The DSI-Utils library (<a href="http://dsiutils.di.unimi.it" class="external-link" target="_blank" rel="nofollow noopener">http://dsiutils.di.unimi.it</a>) offers a class that allows one to reduce the overhead incurred by this many HLL objects through its HyperLogLogCounterArray class. We're struggling with the implementation in Trident though, as the init(Object batchId, TridentCollector collector) method of the aggregator interface does not provide any information about the current "group" the aggregator should be initialized for.</p>
   
   <p>(This was initially posted on Google Groups: <a href="https://groups.google.com/forum/#!topic/storm-user/dthUfkMRNhU" class="external-link" target="_blank" rel="nofollow noopener">https://groups.google.com/forum/#!topic/storm-user/dthUfkMRNhU</a>)</p>
   
   ---
   <details><summary><i>Originally reported by <a title="James Xu" href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=xumingming">xumingming</a>, imported from: <a href="https://issues.apache.org/jira/browse/STORM-7" target="_blank">storm.trident.operation.Aggregator: include group information in init() method</a></i></summary>
   <i><ul>
   <li><b>status</b>: Closed
   <li><b>priority</b>: Minor
   <li><b>resolution</b>: Abandoned
   <li><b>resolved</b>: 2024-07-02T10:54:13+01:00
   <li><b>imported</b>: 2025-01-24
   </ul></i>
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@storm.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [STORM-7] storm.trident.operation.Aggregator: include group information in init() method (storm)

["jira-importer (via GitHub)" <gi...@apache.org>]

jira-importer commented on issue #3849:
URL: https://github.com/apache/storm/issues/3849#issuecomment-2612400049

   <i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=rzo1">rzo1</a>:</i>
   <p>We are in the process of migrating our issue tracking system to GitHub issues. This issue has been stale for the last 5 years, and STORM has evolved with many dependency updates since this issue was last updated.</p>
   
   <p>If the issue persists in the latest version of Storm, please feel free to reopen it or provide new context. Thank you very much!</p>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@storm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org