You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ryanne Dolan <ry...@gmail.com> on 2021/01/02 19:31:26 UTC
Re: When using MM2, how should the consumer be switched from source
to target?
Aki, that's right. Prior to 2.7, you can use the translateOffsets method
from within your client code, or you can write a little command-line tool
to do it. I've done the latter for Cloudera's srm-control tool, as
documented here:
https://docs.cloudera.com/csp/2.0.1/srm-using/topics/srm-migrating-consumer-groups.html
Ryanne
On Thu, Dec 17, 2020, 5:23 PM Aki Yoshida <el...@gmail.com> wrote:
> I'm answering to my own mail and would like to hear if this assumption
> is correct.
> it looks like I need 2.7.0 to have the automatic translation.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
> for earlier versions, there is no way to use the console client but
> use java API RemoteClusterUtils.translateOffsets() to get the offset
> and re-consuming from the new topic.
> Is this correct?
>
> El jue, 17 dic 2020 a las 21:18, Aki Yoshida (<el...@gmail.com>)
> escribió:
> >
> > Hi,
> > I have a question regarding how to migrate a consumer when the
> > subscribing topic has been migrated to the target Kafka broker.
> > Suppose a consumer is consuming from topic1 at the source Kafka
> > broker. When MM2 mirrors this topic using options
> > source.cluster.alias="", replication.policy.separator="", topic named
> > topic1 shows up at the target Kafka broker.
> >
> > When the consumer instance simply switches the bootstrap server to
> > start consuming from this topic1 at the target broker, the consumer
> > seems to start subscribing from the latest offset. I thought the
> > consumer offsets were also migrated to the target and the consumer
> > could simply start subscribing from the mirrored topic from the
> > continued offset. Did I miss some configuration parameters or does the
> > consumer need to perform some actions to be able to consume the
> > records seamlessly?
> >
> > I appreciate for your help. Thanks.
> >
> > regards, aki
>
Re: When using MM2, how should the consumer be switched from source to target?
Posted by Ning Zhang <ni...@gmail.com>.
hello Aki,
I guess overall you are right.
When the consumer have not finish consuming the latest message from primary and switch to the backup, the left messages will be still replicated from primary to backup by MM, so your consumer will still be able to consume the left messages at backup, this is what your commit illustrated.
I think the original test was correct as well, since it assumes no left message in primary before switching consumer to backup.
If you are interested in contributing your commit, it would be great to submit it on https://github.com/apache/kafka/pull/9224, as MM tests could be significantly refactored soon.
Thanks
On 2021/01/13 08:14:06, Aki Yoshida <el...@gmail.com> wrote:
> Hi Ning,
> I mean by "when there are some messages left in the original cluster",
> when the consumer switches from the original cluster to the new
> cluster before entirely consuming the messages at the original
> cluster.
> This commit illustrates this situation.
> https://github.com/elakito/kafka/commit/9063db0f8c4a53f5d9764612af898981d499a7b7
>
> regards, aki
>
> El mié, 13 ene 2021 a las 3:17, Ning Zhang (<ni...@gmail.com>) escribió:
> >
> > Hello Aki,
> >
> > Can you elaborate on "when there are some messages left in the original cluster" ?
> >
> > On 2021/01/12 13:01:40, Aki Yoshida <el...@gmail.com> wrote:
> > > Hi Ryanne,
> > > Thanks for the information regarding the manual translation approach.
> > > But for 2.7.0, I have a question. I thought this translation would
> > > happen implicitly.
> > > I saw test testOneWayReplicationWithAutoOffsetSync of
> > > MirrorConnectorsIntegrationTest that is supposed to test this
> > > behavior. But I have an impression that the consumers can't be
> > > migrated when there are some messages left in the original cluster. I
> > > posted this question to users@kafka a while ago but I didn't receive
> > > any responses.
> > > https://lists.apache.org/thread.html/r0728e9ea89b2713865a33396f108027ac3f0949c2496583532ee963c%40%3Cusers.kafka.apache.org%3E
> > > Could you or someone comment on this?
> > > Thanks.
> > > Regards, aki
> > >
> > > El sáb, 2 ene 2021 a las 20:31, Ryanne Dolan (<ry...@gmail.com>) escribió:
> > > >
> > > > Aki, that's right. Prior to 2.7, you can use the translateOffsets method
> > > > from within your client code, or you can write a little command-line tool
> > > > to do it. I've done the latter for Cloudera's srm-control tool, as
> > > > documented here:
> > > > https://docs.cloudera.com/csp/2.0.1/srm-using/topics/srm-migrating-consumer-groups.html
> > > >
> > > > Ryanne
> > > >
> > > > On Thu, Dec 17, 2020, 5:23 PM Aki Yoshida <el...@gmail.com> wrote:
> > > >
> > > > > I'm answering to my own mail and would like to hear if this assumption
> > > > > is correct.
> > > > > it looks like I need 2.7.0 to have the automatic translation.
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
> > > > > for earlier versions, there is no way to use the console client but
> > > > > use java API RemoteClusterUtils.translateOffsets() to get the offset
> > > > > and re-consuming from the new topic.
> > > > > Is this correct?
> > > > >
> > > > > El jue, 17 dic 2020 a las 21:18, Aki Yoshida (<el...@gmail.com>)
> > > > > escribió:
> > > > > >
> > > > > > Hi,
> > > > > > I have a question regarding how to migrate a consumer when the
> > > > > > subscribing topic has been migrated to the target Kafka broker.
> > > > > > Suppose a consumer is consuming from topic1 at the source Kafka
> > > > > > broker. When MM2 mirrors this topic using options
> > > > > > source.cluster.alias="", replication.policy.separator="", topic named
> > > > > > topic1 shows up at the target Kafka broker.
> > > > > >
> > > > > > When the consumer instance simply switches the bootstrap server to
> > > > > > start consuming from this topic1 at the target broker, the consumer
> > > > > > seems to start subscribing from the latest offset. I thought the
> > > > > > consumer offsets were also migrated to the target and the consumer
> > > > > > could simply start subscribing from the mirrored topic from the
> > > > > > continued offset. Did I miss some configuration parameters or does the
> > > > > > consumer need to perform some actions to be able to consume the
> > > > > > records seamlessly?
> > > > > >
> > > > > > I appreciate for your help. Thanks.
> > > > > >
> > > > > > regards, aki
> > > > >
> > >
>
Re: When using MM2, how should the consumer be switched from source
to target?
Posted by Aki Yoshida <el...@gmail.com>.
Hi Ning,
I mean by "when there are some messages left in the original cluster",
when the consumer switches from the original cluster to the new
cluster before entirely consuming the messages at the original
cluster.
This commit illustrates this situation.
https://github.com/elakito/kafka/commit/9063db0f8c4a53f5d9764612af898981d499a7b7
regards, aki
El mié, 13 ene 2021 a las 3:17, Ning Zhang (<ni...@gmail.com>) escribió:
>
> Hello Aki,
>
> Can you elaborate on "when there are some messages left in the original cluster" ?
>
> On 2021/01/12 13:01:40, Aki Yoshida <el...@gmail.com> wrote:
> > Hi Ryanne,
> > Thanks for the information regarding the manual translation approach.
> > But for 2.7.0, I have a question. I thought this translation would
> > happen implicitly.
> > I saw test testOneWayReplicationWithAutoOffsetSync of
> > MirrorConnectorsIntegrationTest that is supposed to test this
> > behavior. But I have an impression that the consumers can't be
> > migrated when there are some messages left in the original cluster. I
> > posted this question to users@kafka a while ago but I didn't receive
> > any responses.
> > https://lists.apache.org/thread.html/r0728e9ea89b2713865a33396f108027ac3f0949c2496583532ee963c%40%3Cusers.kafka.apache.org%3E
> > Could you or someone comment on this?
> > Thanks.
> > Regards, aki
> >
> > El sáb, 2 ene 2021 a las 20:31, Ryanne Dolan (<ry...@gmail.com>) escribió:
> > >
> > > Aki, that's right. Prior to 2.7, you can use the translateOffsets method
> > > from within your client code, or you can write a little command-line tool
> > > to do it. I've done the latter for Cloudera's srm-control tool, as
> > > documented here:
> > > https://docs.cloudera.com/csp/2.0.1/srm-using/topics/srm-migrating-consumer-groups.html
> > >
> > > Ryanne
> > >
> > > On Thu, Dec 17, 2020, 5:23 PM Aki Yoshida <el...@gmail.com> wrote:
> > >
> > > > I'm answering to my own mail and would like to hear if this assumption
> > > > is correct.
> > > > it looks like I need 2.7.0 to have the automatic translation.
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
> > > > for earlier versions, there is no way to use the console client but
> > > > use java API RemoteClusterUtils.translateOffsets() to get the offset
> > > > and re-consuming from the new topic.
> > > > Is this correct?
> > > >
> > > > El jue, 17 dic 2020 a las 21:18, Aki Yoshida (<el...@gmail.com>)
> > > > escribió:
> > > > >
> > > > > Hi,
> > > > > I have a question regarding how to migrate a consumer when the
> > > > > subscribing topic has been migrated to the target Kafka broker.
> > > > > Suppose a consumer is consuming from topic1 at the source Kafka
> > > > > broker. When MM2 mirrors this topic using options
> > > > > source.cluster.alias="", replication.policy.separator="", topic named
> > > > > topic1 shows up at the target Kafka broker.
> > > > >
> > > > > When the consumer instance simply switches the bootstrap server to
> > > > > start consuming from this topic1 at the target broker, the consumer
> > > > > seems to start subscribing from the latest offset. I thought the
> > > > > consumer offsets were also migrated to the target and the consumer
> > > > > could simply start subscribing from the mirrored topic from the
> > > > > continued offset. Did I miss some configuration parameters or does the
> > > > > consumer need to perform some actions to be able to consume the
> > > > > records seamlessly?
> > > > >
> > > > > I appreciate for your help. Thanks.
> > > > >
> > > > > regards, aki
> > > >
> >
Re: When using MM2, how should the consumer be switched from source to target?
Posted by Ning Zhang <ni...@gmail.com>.
Hello Aki,
Can you elaborate on "when there are some messages left in the original cluster" ?
On 2021/01/12 13:01:40, Aki Yoshida <el...@gmail.com> wrote:
> Hi Ryanne,
> Thanks for the information regarding the manual translation approach.
> But for 2.7.0, I have a question. I thought this translation would
> happen implicitly.
> I saw test testOneWayReplicationWithAutoOffsetSync of
> MirrorConnectorsIntegrationTest that is supposed to test this
> behavior. But I have an impression that the consumers can't be
> migrated when there are some messages left in the original cluster. I
> posted this question to users@kafka a while ago but I didn't receive
> any responses.
> https://lists.apache.org/thread.html/r0728e9ea89b2713865a33396f108027ac3f0949c2496583532ee963c%40%3Cusers.kafka.apache.org%3E
> Could you or someone comment on this?
> Thanks.
> Regards, aki
>
> El sáb, 2 ene 2021 a las 20:31, Ryanne Dolan (<ry...@gmail.com>) escribió:
> >
> > Aki, that's right. Prior to 2.7, you can use the translateOffsets method
> > from within your client code, or you can write a little command-line tool
> > to do it. I've done the latter for Cloudera's srm-control tool, as
> > documented here:
> > https://docs.cloudera.com/csp/2.0.1/srm-using/topics/srm-migrating-consumer-groups.html
> >
> > Ryanne
> >
> > On Thu, Dec 17, 2020, 5:23 PM Aki Yoshida <el...@gmail.com> wrote:
> >
> > > I'm answering to my own mail and would like to hear if this assumption
> > > is correct.
> > > it looks like I need 2.7.0 to have the automatic translation.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
> > > for earlier versions, there is no way to use the console client but
> > > use java API RemoteClusterUtils.translateOffsets() to get the offset
> > > and re-consuming from the new topic.
> > > Is this correct?
> > >
> > > El jue, 17 dic 2020 a las 21:18, Aki Yoshida (<el...@gmail.com>)
> > > escribió:
> > > >
> > > > Hi,
> > > > I have a question regarding how to migrate a consumer when the
> > > > subscribing topic has been migrated to the target Kafka broker.
> > > > Suppose a consumer is consuming from topic1 at the source Kafka
> > > > broker. When MM2 mirrors this topic using options
> > > > source.cluster.alias="", replication.policy.separator="", topic named
> > > > topic1 shows up at the target Kafka broker.
> > > >
> > > > When the consumer instance simply switches the bootstrap server to
> > > > start consuming from this topic1 at the target broker, the consumer
> > > > seems to start subscribing from the latest offset. I thought the
> > > > consumer offsets were also migrated to the target and the consumer
> > > > could simply start subscribing from the mirrored topic from the
> > > > continued offset. Did I miss some configuration parameters or does the
> > > > consumer need to perform some actions to be able to consume the
> > > > records seamlessly?
> > > >
> > > > I appreciate for your help. Thanks.
> > > >
> > > > regards, aki
> > >
>
Re: When using MM2, how should the consumer be switched from source
to target?
Posted by Aki Yoshida <el...@gmail.com>.
Hi Ryanne,
Thanks for the information regarding the manual translation approach.
But for 2.7.0, I have a question. I thought this translation would
happen implicitly.
I saw test testOneWayReplicationWithAutoOffsetSync of
MirrorConnectorsIntegrationTest that is supposed to test this
behavior. But I have an impression that the consumers can't be
migrated when there are some messages left in the original cluster. I
posted this question to users@kafka a while ago but I didn't receive
any responses.
https://lists.apache.org/thread.html/r0728e9ea89b2713865a33396f108027ac3f0949c2496583532ee963c%40%3Cusers.kafka.apache.org%3E
Could you or someone comment on this?
Thanks.
Regards, aki
El sáb, 2 ene 2021 a las 20:31, Ryanne Dolan (<ry...@gmail.com>) escribió:
>
> Aki, that's right. Prior to 2.7, you can use the translateOffsets method
> from within your client code, or you can write a little command-line tool
> to do it. I've done the latter for Cloudera's srm-control tool, as
> documented here:
> https://docs.cloudera.com/csp/2.0.1/srm-using/topics/srm-migrating-consumer-groups.html
>
> Ryanne
>
> On Thu, Dec 17, 2020, 5:23 PM Aki Yoshida <el...@gmail.com> wrote:
>
> > I'm answering to my own mail and would like to hear if this assumption
> > is correct.
> > it looks like I need 2.7.0 to have the automatic translation.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
> > for earlier versions, there is no way to use the console client but
> > use java API RemoteClusterUtils.translateOffsets() to get the offset
> > and re-consuming from the new topic.
> > Is this correct?
> >
> > El jue, 17 dic 2020 a las 21:18, Aki Yoshida (<el...@gmail.com>)
> > escribió:
> > >
> > > Hi,
> > > I have a question regarding how to migrate a consumer when the
> > > subscribing topic has been migrated to the target Kafka broker.
> > > Suppose a consumer is consuming from topic1 at the source Kafka
> > > broker. When MM2 mirrors this topic using options
> > > source.cluster.alias="", replication.policy.separator="", topic named
> > > topic1 shows up at the target Kafka broker.
> > >
> > > When the consumer instance simply switches the bootstrap server to
> > > start consuming from this topic1 at the target broker, the consumer
> > > seems to start subscribing from the latest offset. I thought the
> > > consumer offsets were also migrated to the target and the consumer
> > > could simply start subscribing from the mirrored topic from the
> > > continued offset. Did I miss some configuration parameters or does the
> > > consumer need to perform some actions to be able to consume the
> > > records seamlessly?
> > >
> > > I appreciate for your help. Thanks.
> > >
> > > regards, aki
> >