You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "syeerzy (via GitHub)" <gi...@apache.org> on 2025/11/21 00:50:13 UTC
[I] [Feature] The “First” aggregation type of the aggregation table [doris]
syeerzy opened a new issue, #58204:
URL: https://github.com/apache/doris/issues/58204
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Description
The aggregation table needs to support a First semantics (contrary to Replace aggregation, which retains the latest value, First only retains the first value)
This will be a special aggregation type, because it has the specificity of "never updating", and it will become the only aggregation type that can be used for partitioning. ( Now all aggregation types cannot be used for partitioning)
### Use case
Assuming that we have a batch of click event data in different games from kafka's boundless, we need to aggregate the clicks from the same user and the same game in the same game. The time of a game of different games may range from 10 seconds to hours.
Because the data is boundless and may grow to infinity over time, the data volume of this aggregate table will also increase over time, and the table needs to be partitioned.
In most cases, the event date is a good choice, but a game may take place at 23:59 and last until the next day. We can't predict that there may be other games with a longer game time in the future, so it's not easy to determine the time window to deal with it. Different games may have different rules. The way of expressing a game (self-increasing int, or uuid, etc.) is difficult to unify as the composition of partition rules.
At present, it is impossible to solve this similar scenario (the business can't find a suitable key that can be used for partitioning, and only the aggregate column can be partitioned)
And an aggregate type that never updates will solve this problem.
### Related issues
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Feature] The “First” aggregation type of the aggregation table [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb closed issue #58204: [Feature] The “First” aggregation type of the aggregation table
URL: https://github.com/apache/doris/issues/58204
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Feature] The “First” aggregation type of the aggregation table [doris]
Posted by "AntiTopQuark (via GitHub)" <gi...@apache.org>.
AntiTopQuark commented on issue #58204:
URL: https://github.com/apache/doris/issues/58204#issuecomment-3591585002
> maybe just using `MIN/MAX` to aggregate could work?
The semantics of First, MIN, and MAX are still different; I can implement all three.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Feature] The “First” aggregation type of the aggregation table [doris]
Posted by "AntiTopQuark (via GitHub)" <gi...@apache.org>.
AntiTopQuark commented on issue #58204:
URL: https://github.com/apache/doris/issues/58204#issuecomment-3650666752
Hi, @zclllyybb @syeerzy
Naming the syntax directly as FIRST conflicts with the existing MySQL-style syntax
`ALTER TABLE ${partition_table_name} MODIFY COLUMN name STRING FIRST`,
which moves a column to the first position in the table.
Would it be possible to implement the keyword as FIRST_IF_NOT_NULL instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Feature] The “First” aggregation type of the aggregation table [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on issue #58204:
URL: https://github.com/apache/doris/issues/58204#issuecomment-3683225521
Finally I realized that by https://doris.apache.org/zh-CN/docs/dev/table-design/data-model/aggregate#agg_state we can achieve any aggregate type. seems https://github.com/apache/doris/pull/59017 is not required.
cc @syeerzy @AntiTopQuark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Feature] The “First” aggregation type of the aggregation table [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on issue #58204:
URL: https://github.com/apache/doris/issues/58204#issuecomment-3581987706
maybe just using `MIN/MAX` to aggregate could work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Feature] The “First” aggregation type of the aggregation table [doris]
Posted by "AntiTopQuark (via GitHub)" <gi...@apache.org>.
AntiTopQuark commented on issue #58204:
URL: https://github.com/apache/doris/issues/58204#issuecomment-3565756514
Hi,I want to complete this task. Could you assign the issue to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Feature] The “First” aggregation type of the aggregation table [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on issue #58204:
URL: https://github.com/apache/doris/issues/58204#issuecomment-3685698917
yes, I will close the issue and related pr then. thanks you all @syeerzy @AntiTopQuark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Feature] The “First” aggregation type of the aggregation table [doris]
Posted by "AntiTopQuark (via GitHub)" <gi...@apache.org>.
AntiTopQuark commented on issue #58204:
URL: https://github.com/apache/doris/issues/58204#issuecomment-3684814132
> Finally I realized that by https://doris.apache.org/zh-CN/docs/dev/table-design/data-model/aggregate#agg_state we can achieve any aggregate type. seems [#59017](https://github.com/apache/doris/pull/59017) is not required. cc [@syeerzy](https://github.com/syeerzy) [@AntiTopQuark](https://github.com/AntiTopQuark)
Wow, `AGG_STATE` is truly an extremely powerful feature, and it seems that the 'first' PR isn't really necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org