You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Enis Söztutar <en...@hortonworks.com> on 2014/04/07 12:11:56 UTC
Re: Hadoop Summit EU
Me and Devaraj attended their talk on their solution for paxos based
namenode and HBase replication.
They have two solutions, one for single datacenter, and the other multi DC
geo replication.
For the namenode, there is a wrapper, called ConsistencyNode, that
basically gets the requests, replicate it via their consistency protocol to
other CNodes within the DC (paxos based) in the edit log. If the proposal
for this is accepted, the changes are made durable. However, from my
understanding, on the read side the client chooses only one replica to
read. The client decides to connect to one of the replica namenodes, which
means that it is not doing a paxos read. I think they also wrapped the
client, so that if it gets a FileNotFoundException or something similar, it
will retry on a different server. Also they track the last seen proposal id
as a transaction id for this as well from my understanding (so
read-what-you-write consistency maybe?). The full details of the
consistency was not clear to me from the presentation.
For their multi-DC replication, they are doing a similar thing, but the
data replication is not handled by paxos, only the namenode metadata. For
each datacenter, they have a target replication factor (can be set
differently for each DC, like 0 because of regulatory reasons). The
metadata of NN is replicated via a similar mechanism. The data replication
is async to the metadata replication though. When a block is finalized, the
CNode quorum on that particular DC, schedules a remote copy to one of the
datacenters. That copy job, copies the block with directly writing the
block from the datanode to a remote datanode. Then that remote DC block is
replicated to the target replication by that DC's CNode quorum. When the
target is reached, that DC will create another proposal about the data
replication being complete. So the state machine probably contains where
each data is replicated, but they were still mentioning the client getting
DataNotReplicatedException or something.
Their work on HBase is still WIP. I do not remember much details on the
protocol, except it uses the same replication protocol (their "patented"
paxos based replication).
Of course the devil is in the details. I did not get that from the
presentation.
As a side note, Doug when asked, was saying that they are cooking something
for backups, so maybe their "secret project" also contains multi-DC
consistent state?
Enis
On Sat, Apr 5, 2014 at 1:55 AM, Ted Yu <yu...@gmail.com> wrote:
> Enis:
> There was a talk by Konstantin Boudnik<http://hadoopsummit.org/amsterdam/speakers/#konstantin-boudnik>
> .
>
> Any interesting material from his presentation ?
>
> Cheers
>
Re: Hadoop Summit EU
Posted by Mikhail Antonov <ol...@gmail.com>.
Well...Since that was mentioned anyway, allow me a tiny
correction/clarification.. :)
It's ConsensusNode, not ConsistencyNode, and it's not really custom Paxos
implementation, it's more like interface for coordination service atop
standard NameNode, which may be backed by any consensus library/algorithm,
be it variation of Paxos, ZooKeeper/ZAB, Raft or anything else. The
consensus API itself (ConsensusNode code) and ZooKeeper-based
implementation of consensus protocol is going to be open-sourced (we're
working on it), and once it's out, consensus libraries authors are welcome
to start integration with their libs too.
Regarding HBase - that's actually what's being developed under HBASE-10909,
HBASE-10866 and referenced jiras (everybody interested is welcome to
discuss/feedback).
-Mikhail
2014-04-07 11:36 GMT-07:00 Enis Söztutar <en...@hortonworks.com>:
> Ops sorry this was intented for internal lists. Apologies for any
> confusion.
>
> Enis
>
> On Monday, April 7, 2014, Enis Söztutar <en...@hortonworks.com> wrote:
>
> > Me and Devaraj attended their talk on their solution for paxos based
> > namenode and HBase replication.
> >
> > They have two solutions, one for single datacenter, and the other multi
> DC
> > geo replication.
> >
> > For the namenode, there is a wrapper, called ConsistencyNode, that
> > basically gets the requests, replicate it via their consistency protocol
> to
> > other CNodes within the DC (paxos based) in the edit log. If the proposal
> > for this is accepted, the changes are made durable. However, from my
> > understanding, on the read side the client chooses only one replica to
> > read. The client decides to connect to one of the replica namenodes,
> which
> > means that it is not doing a paxos read. I think they also wrapped the
> > client, so that if it gets a FileNotFoundException or something similar,
> it
> > will retry on a different server. Also they track the last seen proposal
> id
> > as a transaction id for this as well from my understanding (so
> > read-what-you-write consistency maybe?). The full details of the
> > consistency was not clear to me from the presentation.
> > For their multi-DC replication, they are doing a similar thing, but the
> > data replication is not handled by paxos, only the namenode metadata. For
> > each datacenter, they have a target replication factor (can be set
> > differently for each DC, like 0 because of regulatory reasons). The
> > metadata of NN is replicated via a similar mechanism. The data
> replication
> > is async to the metadata replication though. When a block is finalized,
> the
> > CNode quorum on that particular DC, schedules a remote copy to one of the
> > datacenters. That copy job, copies the block with directly writing the
> > block from the datanode to a remote datanode. Then that remote DC block
> is
> > replicated to the target replication by that DC's CNode quorum. When the
> > target is reached, that DC will create another proposal about the data
> > replication being complete. So the state machine probably contains where
> > each data is replicated, but they were still mentioning the client
> getting
> > DataNotReplicatedException or something.
> >
> > Their work on HBase is still WIP. I do not remember much details on the
> > protocol, except it uses the same replication protocol (their "patented"
> > paxos based replication).
> >
> > Of course the devil is in the details. I did not get that from the
> > presentation.
> >
> > As a side note, Doug when asked, was saying that they are cooking
> > something for backups, so maybe their "secret project" also contains
> > multi-DC consistent state?
> >
> > Enis
> >
> >
> > On Sat, Apr 5, 2014 at 1:55 AM, Ted Yu <yuzhihong@gmail.com
> <javascript:_e(%7B%7D,'cvml','yuzhihong@gmail.com');>
> > > wrote:
> >
> >> Enis:
> >> There was a talk by Konstantin Boudnik<
> http://hadoopsummit.org/amsterdam/speakers/#konstantin-boudnik>
> >> .
> >>
> >> Any interesting material from his presentation ?
> >>
> >> Cheers
> >>
> >
> >
>
--
Thanks,
Michael Antonov
Re: Hadoop Summit EU
Posted by Andrew Purtell <ap...@apache.org>.
I'm curious what part of the below communication should restrict it to
private lists?
Anyway, use of dev@hbase for this was fine in my opinion. We try to
discourage discussion on HBase private lists which should really be done on
the dev lists according to the Apache Way.
On Mon, Apr 7, 2014 at 11:36 AM, Enis Söztutar <en...@hortonworks.com> wrote:
> Ops sorry this was intented for internal lists. Apologies for any
> confusion.
>
> Enis
>
> On Monday, April 7, 2014, Enis Söztutar <en...@hortonworks.com> wrote:
>
> > Me and Devaraj attended their talk on their solution for paxos based
> > namenode and HBase replication.
> >
> > They have two solutions, one for single datacenter, and the other multi
> DC
> > geo replication.
> >
> > For the namenode, there is a wrapper, called ConsistencyNode, that
> > basically gets the requests, replicate it via their consistency protocol
> to
> > other CNodes within the DC (paxos based) in the edit log. If the proposal
> > for this is accepted, the changes are made durable. However, from my
> > understanding, on the read side the client chooses only one replica to
> > read. The client decides to connect to one of the replica namenodes,
> which
> > means that it is not doing a paxos read. I think they also wrapped the
> > client, so that if it gets a FileNotFoundException or something similar,
> it
> > will retry on a different server. Also they track the last seen proposal
> id
> > as a transaction id for this as well from my understanding (so
> > read-what-you-write consistency maybe?). The full details of the
> > consistency was not clear to me from the presentation.
> > For their multi-DC replication, they are doing a similar thing, but the
> > data replication is not handled by paxos, only the namenode metadata. For
> > each datacenter, they have a target replication factor (can be set
> > differently for each DC, like 0 because of regulatory reasons). The
> > metadata of NN is replicated via a similar mechanism. The data
> replication
> > is async to the metadata replication though. When a block is finalized,
> the
> > CNode quorum on that particular DC, schedules a remote copy to one of the
> > datacenters. That copy job, copies the block with directly writing the
> > block from the datanode to a remote datanode. Then that remote DC block
> is
> > replicated to the target replication by that DC's CNode quorum. When the
> > target is reached, that DC will create another proposal about the data
> > replication being complete. So the state machine probably contains where
> > each data is replicated, but they were still mentioning the client
> getting
> > DataNotReplicatedException or something.
> >
> > Their work on HBase is still WIP. I do not remember much details on the
> > protocol, except it uses the same replication protocol (their "patented"
> > paxos based replication).
> >
> > Of course the devil is in the details. I did not get that from the
> > presentation.
> >
> > As a side note, Doug when asked, was saying that they are cooking
> > something for backups, so maybe their "secret project" also contains
> > multi-DC consistent state?
> >
> > Enis
> >
> >
> > On Sat, Apr 5, 2014 at 1:55 AM, Ted Yu <yuzhihong@gmail.com
> <javascript:_e(%7B%7D,'cvml','yuzhihong@gmail.com');>
> > > wrote:
> >
> >> Enis:
> >> There was a talk by Konstantin Boudnik<
> http://hadoopsummit.org/amsterdam/speakers/#konstantin-boudnik>
> >> .
> >>
> >> Any interesting material from his presentation ?
> >>
> >> Cheers
> >>
> >
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Hadoop Summit EU
Posted by Andrew Purtell <ap...@apache.org>.
I'm curious what part of the below communication should restrict it to
private lists?
Anyway, use of dev@hbase for this was fine in my opinion. We try to
discourage discussion on HBase private lists which should really be done on
the dev lists according to the Apache Way.
On Mon, Apr 7, 2014 at 11:36 AM, Enis Söztutar <en...@hortonworks.com> wrote:
> Ops sorry this was intented for internal lists. Apologies for any
> confusion.
>
> Enis
>
> On Monday, April 7, 2014, Enis Söztutar <en...@hortonworks.com> wrote:
>
> > Me and Devaraj attended their talk on their solution for paxos based
> > namenode and HBase replication.
> >
> > They have two solutions, one for single datacenter, and the other multi
> DC
> > geo replication.
> >
> > For the namenode, there is a wrapper, called ConsistencyNode, that
> > basically gets the requests, replicate it via their consistency protocol
> to
> > other CNodes within the DC (paxos based) in the edit log. If the proposal
> > for this is accepted, the changes are made durable. However, from my
> > understanding, on the read side the client chooses only one replica to
> > read. The client decides to connect to one of the replica namenodes,
> which
> > means that it is not doing a paxos read. I think they also wrapped the
> > client, so that if it gets a FileNotFoundException or something similar,
> it
> > will retry on a different server. Also they track the last seen proposal
> id
> > as a transaction id for this as well from my understanding (so
> > read-what-you-write consistency maybe?). The full details of the
> > consistency was not clear to me from the presentation.
> > For their multi-DC replication, they are doing a similar thing, but the
> > data replication is not handled by paxos, only the namenode metadata. For
> > each datacenter, they have a target replication factor (can be set
> > differently for each DC, like 0 because of regulatory reasons). The
> > metadata of NN is replicated via a similar mechanism. The data
> replication
> > is async to the metadata replication though. When a block is finalized,
> the
> > CNode quorum on that particular DC, schedules a remote copy to one of the
> > datacenters. That copy job, copies the block with directly writing the
> > block from the datanode to a remote datanode. Then that remote DC block
> is
> > replicated to the target replication by that DC's CNode quorum. When the
> > target is reached, that DC will create another proposal about the data
> > replication being complete. So the state machine probably contains where
> > each data is replicated, but they were still mentioning the client
> getting
> > DataNotReplicatedException or something.
> >
> > Their work on HBase is still WIP. I do not remember much details on the
> > protocol, except it uses the same replication protocol (their "patented"
> > paxos based replication).
> >
> > Of course the devil is in the details. I did not get that from the
> > presentation.
> >
> > As a side note, Doug when asked, was saying that they are cooking
> > something for backups, so maybe their "secret project" also contains
> > multi-DC consistent state?
> >
> > Enis
> >
> >
> > On Sat, Apr 5, 2014 at 1:55 AM, Ted Yu <yuzhihong@gmail.com
> <javascript:_e(%7B%7D,'cvml','yuzhihong@gmail.com');>
> > > wrote:
> >
> >> Enis:
> >> There was a talk by Konstantin Boudnik<
> http://hadoopsummit.org/amsterdam/speakers/#konstantin-boudnik>
> >> .
> >>
> >> Any interesting material from his presentation ?
> >>
> >> Cheers
> >>
> >
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Hadoop Summit EU
Posted by Enis Söztutar <en...@hortonworks.com>.
Ops sorry this was intented for internal lists. Apologies for any
confusion.
Enis
On Monday, April 7, 2014, Enis Söztutar <en...@hortonworks.com> wrote:
> Me and Devaraj attended their talk on their solution for paxos based
> namenode and HBase replication.
>
> They have two solutions, one for single datacenter, and the other multi DC
> geo replication.
>
> For the namenode, there is a wrapper, called ConsistencyNode, that
> basically gets the requests, replicate it via their consistency protocol to
> other CNodes within the DC (paxos based) in the edit log. If the proposal
> for this is accepted, the changes are made durable. However, from my
> understanding, on the read side the client chooses only one replica to
> read. The client decides to connect to one of the replica namenodes, which
> means that it is not doing a paxos read. I think they also wrapped the
> client, so that if it gets a FileNotFoundException or something similar, it
> will retry on a different server. Also they track the last seen proposal id
> as a transaction id for this as well from my understanding (so
> read-what-you-write consistency maybe?). The full details of the
> consistency was not clear to me from the presentation.
> For their multi-DC replication, they are doing a similar thing, but the
> data replication is not handled by paxos, only the namenode metadata. For
> each datacenter, they have a target replication factor (can be set
> differently for each DC, like 0 because of regulatory reasons). The
> metadata of NN is replicated via a similar mechanism. The data replication
> is async to the metadata replication though. When a block is finalized, the
> CNode quorum on that particular DC, schedules a remote copy to one of the
> datacenters. That copy job, copies the block with directly writing the
> block from the datanode to a remote datanode. Then that remote DC block is
> replicated to the target replication by that DC's CNode quorum. When the
> target is reached, that DC will create another proposal about the data
> replication being complete. So the state machine probably contains where
> each data is replicated, but they were still mentioning the client getting
> DataNotReplicatedException or something.
>
> Their work on HBase is still WIP. I do not remember much details on the
> protocol, except it uses the same replication protocol (their "patented"
> paxos based replication).
>
> Of course the devil is in the details. I did not get that from the
> presentation.
>
> As a side note, Doug when asked, was saying that they are cooking
> something for backups, so maybe their "secret project" also contains
> multi-DC consistent state?
>
> Enis
>
>
> On Sat, Apr 5, 2014 at 1:55 AM, Ted Yu <yuzhihong@gmail.com<javascript:_e(%7B%7D,'cvml','yuzhihong@gmail.com');>
> > wrote:
>
>> Enis:
>> There was a talk by Konstantin Boudnik<http://hadoopsummit.org/amsterdam/speakers/#konstantin-boudnik>
>> .
>>
>> Any interesting material from his presentation ?
>>
>> Cheers
>>
>
>