You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 06:13:45 UTC
[I] Add record count payload to support pv/uv [hudi]
hudi-bot opened a new issue, #15638:
URL: https://github.com/apache/hudi/issues/15638
In the past, pv/uv was processed through flink + window aggregation. This method has the risk of delayed data discarding and state explosion. We use record count payload without these risks.
In order to use 'RecordCountAvroPayload', we need to add field [hoodie_record_count bigint] to the schema when creating the hudi table to record the result of pv/uv, field 'hoodie_record_count' does not need to be filled, and flink will automatically set it to "null", "null" represents 1
eg:
Order field is 'ts', schema is :
{[
{"name":"id","type":"string"}
,
\{"name":"ts","type":"long"},
\{"name":"name","type":"string"},
\{"name":"hoodie_record_count","type":"long"}
]}
case 1
Current data:
id ts name hoodie_record_count
1 1 name_1 1
Insert data:
id ts name hoodie_record_count
1 2 name_2 2
Result data:
id ts name hoodie_record_count
1 2 name_2 3
case 2
Current data:
id ts name hoodie_record_count
1 2 name_1 null
Insert data:
id ts name hoodie_record_count
1 1 name_2 1
Result data:
id ts name hoodie_record_count
1 2 name_1 2
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5413
- Type: New Feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add record count payload to support pv/uv [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #15638:
URL: https://github.com/apache/hudi/issues/15638#issuecomment-3630122581
### Linked PR(s)
- https://github.com/apache/hudi/pull/7499
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org