You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn> on 2019/10/09 05:58:40 UTC
Can CaptureChangeMySQL be scheduled to all nodes instead of primary node?
I am using CaptureChangeMySQL to replicate the database.
There are many data sources and so there're many CaptureChangeMySQL processors.
The CaptureChangeMySQL throws same slave id error if scheduled on all nodes. So it can only be scheduled on primary node. This causes very heavy load on the primary node.
Is there any method than i can distribute the CaptureChangeMySQL processors to all nodes instead of only to primary node?
Thanks,
Lei
wanglei2@geekplus.com.cn
Re: Can CaptureChangeMySQL be scheduled to all nodes instead of
primary node?
Posted by Koji Kawamura <ij...@gmail.com>.
Hi Lei,
I don't know any NiFi built-in feature to achieve that.
To distribute CaptureChangeMySQL load among nodes, I'd deploy separate
standalone NiFi (or even MiNiFi Java) in addition to the main NiFi
cluster for the main data flow.
For example, if there are 5 databases and 3 NiFi nodes, deploy a 3
node NiFi cluster with an InputPort.
And also, run standalone NiFi/MiNiFi processes on each node, too,
node-a (datasource 1 and 2), node-b (datasource 3 and 4), node-c
(datasource 5) then use RemoteProcessGroup to send captured data to
the main NiFi cluster.
This approach may be harder to maintain, but feasible.
Thanks,
Koji
On Wed, Oct 9, 2019 at 3:06 PM wanglei2@geekplus.com.cn
<wa...@geekplus.com.cn> wrote:
>
> I am using CaptureChangeMySQL to replicate the database.
> There are many data sources and so there're many CaptureChangeMySQL processors.
> The CaptureChangeMySQL throws same slave id error if scheduled on all nodes. So it can only be scheduled on primary node. This causes very heavy load on the primary node.
>
> Is there any method than i can distribute the CaptureChangeMySQL processors to all nodes instead of only to primary node?
>
> Thanks,
> Lei
>
> ________________________________
> wanglei2@geekplus.com.cn
Re: Can CaptureChangeMySQL be scheduled to all nodes instead of
primary node?
Posted by Koji Kawamura <ij...@gmail.com>.
Hi Lei,
I don't know any NiFi built-in feature to achieve that.
To distribute CaptureChangeMySQL load among nodes, I'd deploy separate
standalone NiFi (or even MiNiFi Java) in addition to the main NiFi
cluster for the main data flow.
For example, if there are 5 databases and 3 NiFi nodes, deploy a 3
node NiFi cluster with an InputPort.
And also, run standalone NiFi/MiNiFi processes on each node, too,
node-a (datasource 1 and 2), node-b (datasource 3 and 4), node-c
(datasource 5) then use RemoteProcessGroup to send captured data to
the main NiFi cluster.
This approach may be harder to maintain, but feasible.
Thanks,
Koji
On Wed, Oct 9, 2019 at 3:06 PM wanglei2@geekplus.com.cn
<wa...@geekplus.com.cn> wrote:
>
> I am using CaptureChangeMySQL to replicate the database.
> There are many data sources and so there're many CaptureChangeMySQL processors.
> The CaptureChangeMySQL throws same slave id error if scheduled on all nodes. So it can only be scheduled on primary node. This causes very heavy load on the primary node.
>
> Is there any method than i can distribute the CaptureChangeMySQL processors to all nodes instead of only to primary node?
>
> Thanks,
> Lei
>
> ________________________________
> wanglei2@geekplus.com.cn