You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ratis.apache.org by Alan WU <al...@gmail.com> on 2018/12/12 11:16:21 UTC
Two questions about Ratis
Hello developers,
It's nice to meet you, I'm glad to hear that you are building a middleware
of open source project about Raft protocol, I've forked the code into my
code repository, I really appreciate your contribution to the open source
community.
There are some questions made me confused
1. I created a class and extend to BaseStateMachine, and override the
method applyTransaction , below is the method code:
@Override
public CompletableFuture<Message> applyTransaction(TransactionContext trx) {
final LogEntryProto entry = trx.getLogEntry();
ByteString data = entry.getSmLogEntry().getData();
final long index = entry.getIndex();
byte[] bytes = data.toByteArray();
String log = new String(bytes,Charset.forName("UTF-8"));
System.out.println("applyTransaction::"+index);
if(replicationTask.executeTransaction(log)){
updateLastAppliedTermIndex(entry.getTerm(), index);
return CompletableFuture.completedFuture(Message.valueOf("ok"));
}
return CompletableFuture.completedFuture(Message.valueOf("failed"));
}
My transaction is used to create or update a record into Mysql db, so the
entry log can be seen as a SQL bin log, so my question is, how to guarantee
my database record won't be duplicated when the machine is restarted and
then the snapshot replay.
2. how to physically delete the history logs when all machines SQL log is
executed successfully.
Looking forward to your reply.
Best regards,
Alan
Re: Two questions about Ratis
Posted by Tsz Wo Sze <sz...@gmail.com>.
Yes, we will add of feature deleting old logs soon.
BTW, thanks for using Ratis!
Tsz-Wo
On Thu, Dec 20, 2018 at 1:50 PM Alan WU <al...@gmail.com> wrote:
>
> Thank you very much for answering my questions, for the second question, do
> you have any plan to add this feature in Ratis?
>
> Regards,
> Alan
>
> On Tue, Dec 18, 2018 at 10:53 AM Tsz Wo Sze <sz...@gmail.com> wrote:
>
> > Hi Alan,
> >
> > 1. This is a WAL (Write Ahead Log) question, i.e. how a WAL prevents
> > duplication. You need to solve this problem even with a single server
> > so that it is not specific to Ratis/Raft. One way is to make the db
> > transactions idempotent. Another way to use a pair of begin-end
> > transactions.
> >
> > 2. This is currently a missing feature in Ratis.
> >
> > Please feel free to send you questions to
> > user@ratis.incubator.apache.org (instead of dev). Thanks.
> >
> > Tsz-Wo
> >
> > On Thu, Dec 13, 2018 at 7:35 AM Alan WU <al...@gmail.com> wrote:
> > >
> > > Hello developers,
> > > It's nice to meet you, I'm glad to hear that you are building a
> > middleware
> > > of open source project about Raft protocol, I've forked the code into my
> > > code repository, I really appreciate your contribution to the open source
> > > community.
> > >
> > > There are some questions made me confused
> > >
> > > 1. I created a class and extend to BaseStateMachine, and override the
> > > method applyTransaction , below is the method code:
> > >
> > > @Override
> > > public CompletableFuture<Message> applyTransaction(TransactionContext
> > trx) {
> > > final LogEntryProto entry = trx.getLogEntry();
> > > ByteString data = entry.getSmLogEntry().getData();
> > > final long index = entry.getIndex();
> > > byte[] bytes = data.toByteArray();
> > > String log = new String(bytes,Charset.forName("UTF-8"));
> > > System.out.println("applyTransaction::"+index);
> > > if(replicationTask.executeTransaction(log)){
> > > updateLastAppliedTermIndex(entry.getTerm(), index);
> > > return CompletableFuture.completedFuture(Message.valueOf("ok"));
> > > }
> > > return CompletableFuture.completedFuture(Message.valueOf("failed"));
> > > }
> > >
> > > My transaction is used to create or update a record into Mysql db, so the
> > > entry log can be seen as a SQL bin log, so my question is, how to
> > guarantee
> > > my database record won't be duplicated when the machine is restarted and
> > > then the snapshot replay.
> > >
> > > 2. how to physically delete the history logs when all machines SQL log is
> > > executed successfully.
> > >
> > > Looking forward to your reply.
> > >
> > > Best regards,
> > > Alan
> >
Re: Two questions about Ratis
Posted by Alan WU <al...@gmail.com>.
Thank you very much for answering my questions, for the second question, do
you have any plan to add this feature in Ratis?
Regards,
Alan
On Tue, Dec 18, 2018 at 10:53 AM Tsz Wo Sze <sz...@gmail.com> wrote:
> Hi Alan,
>
> 1. This is a WAL (Write Ahead Log) question, i.e. how a WAL prevents
> duplication. You need to solve this problem even with a single server
> so that it is not specific to Ratis/Raft. One way is to make the db
> transactions idempotent. Another way to use a pair of begin-end
> transactions.
>
> 2. This is currently a missing feature in Ratis.
>
> Please feel free to send you questions to
> user@ratis.incubator.apache.org (instead of dev). Thanks.
>
> Tsz-Wo
>
> On Thu, Dec 13, 2018 at 7:35 AM Alan WU <al...@gmail.com> wrote:
> >
> > Hello developers,
> > It's nice to meet you, I'm glad to hear that you are building a
> middleware
> > of open source project about Raft protocol, I've forked the code into my
> > code repository, I really appreciate your contribution to the open source
> > community.
> >
> > There are some questions made me confused
> >
> > 1. I created a class and extend to BaseStateMachine, and override the
> > method applyTransaction , below is the method code:
> >
> > @Override
> > public CompletableFuture<Message> applyTransaction(TransactionContext
> trx) {
> > final LogEntryProto entry = trx.getLogEntry();
> > ByteString data = entry.getSmLogEntry().getData();
> > final long index = entry.getIndex();
> > byte[] bytes = data.toByteArray();
> > String log = new String(bytes,Charset.forName("UTF-8"));
> > System.out.println("applyTransaction::"+index);
> > if(replicationTask.executeTransaction(log)){
> > updateLastAppliedTermIndex(entry.getTerm(), index);
> > return CompletableFuture.completedFuture(Message.valueOf("ok"));
> > }
> > return CompletableFuture.completedFuture(Message.valueOf("failed"));
> > }
> >
> > My transaction is used to create or update a record into Mysql db, so the
> > entry log can be seen as a SQL bin log, so my question is, how to
> guarantee
> > my database record won't be duplicated when the machine is restarted and
> > then the snapshot replay.
> >
> > 2. how to physically delete the history logs when all machines SQL log is
> > executed successfully.
> >
> > Looking forward to your reply.
> >
> > Best regards,
> > Alan
>
Re: Two questions about Ratis
Posted by Tsz Wo Sze <sz...@gmail.com>.
Hi Alan,
1. This is a WAL (Write Ahead Log) question, i.e. how a WAL prevents
duplication. You need to solve this problem even with a single server
so that it is not specific to Ratis/Raft. One way is to make the db
transactions idempotent. Another way to use a pair of begin-end
transactions.
2. This is currently a missing feature in Ratis.
Please feel free to send you questions to
user@ratis.incubator.apache.org (instead of dev). Thanks.
Tsz-Wo
On Thu, Dec 13, 2018 at 7:35 AM Alan WU <al...@gmail.com> wrote:
>
> Hello developers,
> It's nice to meet you, I'm glad to hear that you are building a middleware
> of open source project about Raft protocol, I've forked the code into my
> code repository, I really appreciate your contribution to the open source
> community.
>
> There are some questions made me confused
>
> 1. I created a class and extend to BaseStateMachine, and override the
> method applyTransaction , below is the method code:
>
> @Override
> public CompletableFuture<Message> applyTransaction(TransactionContext trx) {
> final LogEntryProto entry = trx.getLogEntry();
> ByteString data = entry.getSmLogEntry().getData();
> final long index = entry.getIndex();
> byte[] bytes = data.toByteArray();
> String log = new String(bytes,Charset.forName("UTF-8"));
> System.out.println("applyTransaction::"+index);
> if(replicationTask.executeTransaction(log)){
> updateLastAppliedTermIndex(entry.getTerm(), index);
> return CompletableFuture.completedFuture(Message.valueOf("ok"));
> }
> return CompletableFuture.completedFuture(Message.valueOf("failed"));
> }
>
> My transaction is used to create or update a record into Mysql db, so the
> entry log can be seen as a SQL bin log, so my question is, how to guarantee
> my database record won't be duplicated when the machine is restarted and
> then the snapshot replay.
>
> 2. how to physically delete the history logs when all machines SQL log is
> executed successfully.
>
> Looking forward to your reply.
>
> Best regards,
> Alan