You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/09/17 15:55:42 UTC
[GitHub] [druid] OurNewestMember opened a new issue, #13108: [Draft] feature request: add aggregator finalization to ingest
OurNewestMember opened a new issue, #13108:
URL: https://github.com/apache/druid/issues/13108
The objective is for ingestion tasks to produce segments which can contain finalized aggregations. This can eliminate the need for a an extra step (a query to produce finalized aggregations) to use the column as a primitive value.
Example 1:
- currently (as of around druid 0.23) realtime ingest using a stringLast aggregator should produce a column with a complex data type
- To retrieve the primitive string value, the column values would need to be aggregated in a query with finalization
Questions/etc:
- Would this feature require an additional ingest step such as a merge?
- Additional consequences of this? (eg, could it open the door for perfect rollup/non-dynamic partitioning in realtime ingests?)
- Would there need to be a way to force merging to ensure aggregator finalization when it might not otherwise be executed?
- Should intermediate persists and even handed off segments remain unfinalized?
- Could this be abstracted to work for batch ingests (indexing and compaction) and streaming ingests?
- Obviously one tricky aspect is that once the aggregation is finalized, the value/column generally loses the aggregation's original semantics (eg, may no longer be combined with other finalized or unfinalized values using the same aggregator type and settings)
- eg, after finalizing some value `{"lhs":123,"rhs":"myStringLastValue"}` to `"myStringLastValue"`, the value could be combined with another stringLast value (finalized or unfinalized) but might require using the time value from the `__time` column which may not have been the parameter used to create the original unfinalized value in the first place -- ie, the semantics for performing "another operation" on the column do not necessarily work the same as they would have without the additional finalization operation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] [Draft] feature request: add aggregator finalization to ingest (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13108:
URL: https://github.com/apache/druid/issues/13108#issuecomment-1933154365
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] [Draft] feature request: add aggregator finalization to ingest (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13108:
URL: https://github.com/apache/druid/issues/13108#issuecomment-1883994830
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] [Draft] feature request: add aggregator finalization to ingest (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #13108: [Draft] feature request: add aggregator finalization to ingest
URL: https://github.com/apache/druid/issues/13108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org