You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn> on 2019/07/26 02:36:50 UTC
CaptureChangeMySQL throws ServerException when configured with all nodes execution
If I config the Scheduling Execution to All nodes, there will be Exception in the log:
com.github.shyiko.mysql.binlog.network.ServerException: A slave with the same server_uuid/server_id as this slave has conne
cted to the master; the first event 'mysql-bin.000502' at 298034601, the last event read from './mysql-bin.000502' at 29865
2073, the last byte read from './mysql-bin.000502' at 298652073.
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:882)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:559)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:793)
at java.lang.Thread.run(Thread.java:745)
Does it affect the accuracy of the CaptureChangeMySQL result?
wanglei2@geekplus.com.cn
Re: Re: CaptureChangeMySQL throws ServerException when configured with all nodes execution
Posted by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>.
Thanks， got it
wanglei2@geekplus.com.cn
From: Purushotham Pushpavanthar
Date: 2019-07-30 04:10
To: users
Subject: Re: Re: CaptureChangeMySQL throws ServerException when configured with all nodes execution
Yes, CaptureChangeMySQL processor is designed run on Primary node which gives HA to the processor in a cluster.
The CaptureChangeMySQL runs on a single thread but consumes JVM memory based on load from source MySQL server.
Running multiple CaptureChangeMySQL processors connecting to different MySQL servers shouldn't be a problem if the buffer queue size and queue offer timeouts are configured correctly.
No, there is no other way to configure them to run on different nodes in a cluster. If that is what you want, then you can have one standalone node running for each CaptureChangeMySQL processor.
Regards,
Purushotham Pushpavanth
On Sun, 28 Jul 2019 at 16:08, wanglei2@geekplus.com.cn <wa...@geekplus.com.cn> wrote:
So the CaptureChangeMySQL processor is designed to be configured as run on Primary node? even my nifi is cluster mode.
Now I have many CaptureChangeMySQL processors, connecting to different Mysql Server. If all run on the primary node, the primary node is too high loaded.
Is there any method to distribute the different CaptureChangeMySQL processors to different nodes?
Thanks
wanglei2@geekplus.com.cn
From: Purushotham Pushpavanthar
Date: 2019-07-28 16:33
To: users
Subject: Re: CaptureChangeMySQL throws ServerException when configured with all nodes execution
Hi,
If CaptureChangeMySQL is configured to run on all nodes, there will duplicate events for sure.
However, in your case the exception is thrown for having same "Sever ID" for all instances of the processor connecting to MySQL server.
I don't see any data loss due currecurrent execution.
Regards,
Purushotham Pushpavanth
On Fri, 26 Jul 2019 at 08:07, wanglei2@geekplus.com.cn <wa...@geekplus.com.cn> wrote:
If I config the Scheduling Execution to All nodes, there will be Exception in the log:
com.github.shyiko.mysql.binlog.network.ServerException: A slave with the same server_uuid/server_id as this slave has conne
cted to the master; the first event 'mysql-bin.000502' at 298034601, the last event read from './mysql-bin.000502' at 29865
2073, the last byte read from './mysql-bin.000502' at 298652073.
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:882)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:559)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:793)
at java.lang.Thread.run(Thread.java:745)
Does it affect the accuracy of the CaptureChangeMySQL result?
wanglei2@geekplus.com.cn
Re: Re: CaptureChangeMySQL throws ServerException when configured
with all nodes execution
Posted by Purushotham Pushpavanthar <pu...@gmail.com>.
Yes, CaptureChangeMySQL processor is designed run on Primary node which
gives HA to the processor in a cluster.
The CaptureChangeMySQL runs on a single thread but consumes JVM memory
based on load from source MySQL server.
Running multiple CaptureChangeMySQL processors connecting to different
MySQL servers shouldn't be a problem if the buffer queue size and queue
offer timeouts are configured correctly.
No, there is no other way to configure them to run on different nodes in a
cluster. If that is what you want, then you can have one standalone node
running for each CaptureChangeMySQL processor.
Regards,
Purushotham Pushpavanth
On Sun, 28 Jul 2019 at 16:08, wanglei2@geekplus.com.cn <
wanglei2@geekplus.com.cn> wrote:
>
> So the CaptureChangeMySQL processor is designed to be configured as run
> on Primary node? even my nifi is cluster mode.
> Now I have many CaptureChangeMySQL processors, connecting to
> different Mysql Server. If all run on the primary node, the primary node
> is too high loaded.
> Is there any method to distribute the different CaptureChangeMySQL
> processors to different nodes?
>
> Thanks
> ------------------------------
> wanglei2@geekplus.com.cn
>
>
> *From:* Purushotham Pushpavanthar <pu...@gmail.com>
> *Date:* 2019-07-28 16:33
> *To:* users <us...@nifi.apache.org>
> *Subject:* Re: CaptureChangeMySQL throws ServerException when configured
> with all nodes execution
> Hi,
>
> If CaptureChangeMySQL is configured to run on all nodes, there will
> duplicate events for sure.
> However, in your case the exception is thrown for having same "Sever ID"
> for all instances of the processor connecting to MySQL server.
> I don't see any data loss due currecurrent execution.
>
> Regards,
> Purushotham Pushpavanth
>
>
>
> On Fri, 26 Jul 2019 at 08:07, wanglei2@geekplus.com.cn <
> wanglei2@geekplus.com.cn> wrote:
>
>>
>> If I config the Scheduling Execution to All nodes, there will be
>> Exception in the log:
>>
>> com.github.shyiko.mysql.binlog.network.ServerException: A slave with the
>> same server_uuid/server_id as this slave has conne
>> cted to the master; the first event 'mysql-bin.000502' at 298034601, the
>> last event read from './mysql-bin.000502' at 29865
>> 2073, the last byte read from './mysql-bin.000502' at 298652073.
>> at
>> com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:882)
>> at
>> com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:559)
>> at
>> com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:793)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Does it affect the accuracy of the CaptureChangeMySQL result?
>>
>> ------------------------------
>> wanglei2@geekplus.com.cn
>>
>
Re: Re: CaptureChangeMySQL throws ServerException when configured with all nodes execution
Posted by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>.
So the CaptureChangeMySQL processor is designed to be configured as run on Primary node? even my nifi is cluster mode.
Now I have many CaptureChangeMySQL processors, connecting to different Mysql Server. If all run on the primary node, the primary node is too high loaded.
Is there any method to distribute the different CaptureChangeMySQL processors to different nodes?
Thanks
wanglei2@geekplus.com.cn
From: Purushotham Pushpavanthar
Date: 2019-07-28 16:33
To: users
Subject: Re: CaptureChangeMySQL throws ServerException when configured with all nodes execution
Hi,
If CaptureChangeMySQL is configured to run on all nodes, there will duplicate events for sure.
However, in your case the exception is thrown for having same "Sever ID" for all instances of the processor connecting to MySQL server.
I don't see any data loss due currecurrent execution.
Regards,
Purushotham Pushpavanth
On Fri, 26 Jul 2019 at 08:07, wanglei2@geekplus.com.cn <wa...@geekplus.com.cn> wrote:
If I config the Scheduling Execution to All nodes, there will be Exception in the log:
com.github.shyiko.mysql.binlog.network.ServerException: A slave with the same server_uuid/server_id as this slave has conne
cted to the master; the first event 'mysql-bin.000502' at 298034601, the last event read from './mysql-bin.000502' at 29865
2073, the last byte read from './mysql-bin.000502' at 298652073.
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:882)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:559)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:793)
at java.lang.Thread.run(Thread.java:745)
Does it affect the accuracy of the CaptureChangeMySQL result?
wanglei2@geekplus.com.cn
Re: CaptureChangeMySQL throws ServerException when configured with
all nodes execution
Posted by Purushotham Pushpavanthar <pu...@gmail.com>.
Hi,
If CaptureChangeMySQL is configured to run on all nodes, there will
duplicate events for sure.
However, in your case the exception is thrown for having same "Sever ID"
for all instances of the processor connecting to MySQL server.
I don't see any data loss due currecurrent execution.
Regards,
Purushotham Pushpavanth
On Fri, 26 Jul 2019 at 08:07, wanglei2@geekplus.com.cn <
wanglei2@geekplus.com.cn> wrote:
>
> If I config the Scheduling Execution to All nodes, there will be Exception
> in the log:
>
> com.github.shyiko.mysql.binlog.network.ServerException: A slave with the
> same server_uuid/server_id as this slave has conne
> cted to the master; the first event 'mysql-bin.000502' at 298034601, the
> last event read from './mysql-bin.000502' at 29865
> 2073, the last byte read from './mysql-bin.000502' at 298652073.
> at
> com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:882)
> at
> com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:559)
> at
> com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:793)
> at java.lang.Thread.run(Thread.java:745)
>
> Does it affect the accuracy of the CaptureChangeMySQL result?
>
> ------------------------------
> wanglei2@geekplus.com.cn
>