You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by GitBox <gi...@apache.org> on 2022/01/24 08:31:18 UTC
[GitHub] [incubator-pegasus] Shuo-Jia opened a new issue #892: Feature: Enhance the ease of use and efficiency of duplication
Shuo-Jia opened a new issue #892:
URL: https://github.com/apache/incubator-pegasus/issues/892
## Feature Request
Duplication has some shortcoming：
- **It depends remote filesystem to sync the checkpoint:**
It leads to complexity in use, which make you have to support HDFS or FDS, and must start duplicate in three steps: update checkpoint in master, restore checkpoint in follower and start sync plog data in master.
- **The synchronization of plog data only sends a single mutation at each RPC:**
The rate of sync is slow and cause the plog pile up in sometimes.
The issue request will hope enhance it:
- Use `learn` to replace `backup-restore` to sync checkpoint
- Use `batch-send` to replace `single-send` to sync plog
- Connect the above two processes to automate them, and the establishment of duplication can be completed with one command
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] Shuo-Jia commented on issue #892: Feature: Enhance the ease of use and efficiency of duplication
Posted by GitBox <gi...@apache.org>.
Shuo-Jia commented on issue #892:
URL: https://github.com/apache/incubator-pegasus/issues/892#issuecomment-1019923010
### The flow chart is as follow:
1. master meta: receive create dup table request and mark current `max_plog_commit_decree` as start plog point
2. master meta: init the dup status as `DS_PREPARE`, which means master replica need prepare latest_checkpoint for duplicating
3. master replica: generate latest checkpoint , which require `laster_durable_decee >= max_plog_commit_decree`
4. master replica: update all checkpoint has been prepared and response master meta
5. master meta: change to `DS_APP`, which means all checkpoints have prepared and follower can duplicate it
6. master meta: send create table request to follower meta, and require the table init with `master replica checkpoint`(like restore progress which init table with `backup replica checkpoint`)
7. follower meta: create dup table with dup info
8. follower replica: start open or init replica
9. follower replica: duplicate checkpoint from master replica before init replica, and load replica using duplicated data
10. master meta: check the follower table if create completed periodically
11. master meta: if completed, change to `DS_LOG`, which means master replica can send plog data to follower table
12. master replica: start pipeline to batch send plog data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] Shuo-Jia edited a comment on issue #892: Feature: Enhance the ease of use and efficiency of duplication
Posted by GitBox <gi...@apache.org>.
Shuo-Jia edited a comment on issue #892:
URL: https://github.com/apache/incubator-pegasus/issues/892#issuecomment-1019923010
### The flow chart is as follow:
#### NOTE: The implementation refactor previous `DS_START` to three stage: `DS_PREPARE`, `DS_APP`, `DS_LOG`, and will be explained in follow descreption
1. master meta: receive create dup table request and mark current `max_plog_commit_decree` as start plog point
2. master meta: init the dup status as `DS_PREPARE`, which means master replica need prepare latest_checkpoint for duplicating
3. master replica: generate latest checkpoint , which require `laster_durable_decee >= max_plog_commit_decree`
4. master replica: update all checkpoint has been prepared and response master meta
5. master meta: change to `DS_APP`, which means all checkpoints have prepared and follower can duplicate it
6. master meta: send create table request to follower meta, and require the table init with `master replica checkpoint`(like restore progress which init table with `backup replica checkpoint`)
7. follower meta: create dup table with dup info
8. follower replica: start open or init replica
9. follower replica: duplicate checkpoint from master replica before init replica, and load replica using duplicated data
10. master meta: check the follower table if create completed periodically
11. master meta: if completed, change to `DS_LOG`, which means master replica can send plog data to follower table
12. master replica: start pipeline to batch send plog data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] cauchy1988 commented on issue #892: Feature: Enhance the ease of use and efficiency of duplication
Posted by GitBox <gi...@apache.org>.
cauchy1988 commented on issue #892:
URL: https://github.com/apache/incubator-pegasus/issues/892#issuecomment-1021271499
although, I'm not so familiar with the whole pegasus system, but according to your descriptions, I think the original design is easier to realize but hard or inconvenient to use, on the other hand, the new design i think is a little complex to implement because you may handle more possible corner case; hower the new design will make ’duplication-usage‘ more friendly and easy to use， looking forward to your awesome implementation~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] neverchanje commented on issue #892: Feature: Enhance the ease of use and efficiency of duplication
Posted by GitBox <gi...@apache.org>.
neverchanje commented on issue #892:
URL: https://github.com/apache/incubator-pegasus/issues/892#issuecomment-1020795505
The motivation is fair, lgtm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] neverchanje commented on issue #892: Feature: Enhance the ease of use and efficiency of duplication
Posted by GitBox <gi...@apache.org>.
neverchanje commented on issue #892:
URL: https://github.com/apache/incubator-pegasus/issues/892#issuecomment-1020795505
The motivation is fair, lgtm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] Shuo-Jia edited a comment on issue #892: Feature: Enhance the ease of use and efficiency of duplication
Posted by GitBox <gi...@apache.org>.
Shuo-Jia edited a comment on issue #892:
URL: https://github.com/apache/incubator-pegasus/issues/892#issuecomment-1019923010
### The flow chart is as follow:
#### NOTE: The implementation refactor previous `DS_START` to three stage: `DS_PREPARE`, `DS_APP`, `DS_LOG`, and will be explained in follow descreption
1. **master meta**: receive create dup table request and mark current `max_plog_commit_decree` as start plog point
2. **master meta**: init the dup status as `DS_PREPARE`, which means master replica need prepare latest_checkpoint for duplicating
3. **master replica**: generate latest checkpoint , which require `laster_durable_decee >= max_plog_commit_decree`
4. **master replica**: update all checkpoint has been prepared and response master meta
5. **master meta**: change to `DS_APP`, which means all checkpoints have prepared and follower can duplicate it
6. **master meta**: send create table request to follower meta, and require the table init with `master replica checkpoint`(like restore progress which init table with `backup replica checkpoint`)
7. **follower meta**: create dup table with dup info
8. **follower replica**: start open or init replica
9. **follower replica**: duplicate checkpoint from master replica before init replica, and load replica using duplicated data
10. **master meta**: check the follower table if create completed periodically
11. **master meta**: if completed, change to `DS_LOG`, which means master replica can send plog data to follower table
12. **master replica**: start pipeline to batch send plog data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] cauchy1988 commented on issue #892: Feature: Enhance the ease of use and efficiency of duplication
Posted by GitBox <gi...@apache.org>.
cauchy1988 commented on issue #892:
URL: https://github.com/apache/incubator-pegasus/issues/892#issuecomment-1021271499
although, I'm not so familiar with the whole pegasus system, but according to your descriptions, I think the original design is easier to realize but hard or inconvenient to use, on the other hand, the new design i think is a little complex to implement because you may handle more possible corner case; hower the new design will make ’duplication-usage‘ more friendly and easy to use， looking forward to your awesome implementation~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org