You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "gosonzhang (via GitHub)" <gi...@apache.org> on 2024/10/23 01:52:33 UTC
[PR] [INLONG-11386][TubeMQ] Use local files to save consumer group offset information [inlong]
gosonzhang opened a new pull request, #11395:
URL: https://github.com/apache/inlong/pull/11395
Fixes #11386
Optimize content:
1. Broker regularly refreshes the consumer group offset information in memory to the file Offset Cache according to the configured offsetStgCacheFlushMs value;
2. The consumer group offset information stored in the file Offset Cache is regularly written to the offsetStgFilePath/offsetDir directory according to the configured offsetStgFileSyncMs value. The file name successfully saved is offsets.meta. The persistence logic is as follows:
- Persistently save the consumer group offset information to the file offsets.tmp. If the write fails, output the failure log, alarm and exit the persistence process;
- Read the offsets.tmp file and compare it with the previously written information to see if it is consistent. If it is inconsistent, output the failure log and exit the persistence process;
- Migrate the file from offsets.tmp to offsets.mid. If the offsets.meta file currently exists, change offsets.meta to offsets.old;
- Migrate the file from offsets.mid to offsets.meta, and delete the offsets.old file at the same time.
3. Broker supports saving the current consumer group's Offset collection information to the backupPath path specified in the API interface through the HTTP API admin_backup_group_offsets. The saved file name is offsets.meta.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-11386][TubeMQ] Use local files to save consumer group offset information [inlong]
Posted by "gosonzhang (via GitHub)" <gi...@apache.org>.
gosonzhang merged PR #11395:
URL: https://github.com/apache/inlong/pull/11395
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-11386][TubeMQ] Use local files to save consumer group offset information [inlong]
Posted by "aloyszhang (via GitHub)" <gi...@apache.org>.
aloyszhang commented on code in PR #11395:
URL: https://github.com/apache/inlong/pull/11395#discussion_r1811952473
##########
inlong-tubemq/tubemq-core/src/main/java/org/apache/inlong/tubemq/corebase/TBaseConstants.java:
##########
@@ -84,4 +84,9 @@ public class TBaseConstants {
public static final long CFG_MIN_META_FORCE_UPDATE_PERIOD = 1 * 60 * 1000;
public static final long CFG_STATS_MIN_SNAPSHOT_PERIOD_MS = 2000;
+ public static final int OFFSET_HISTORY_RECORD_VERSION = 1;
Review Comment:
Not used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org