Move Storm to new data center

[Binh Nguyen Van <bi...@gmail.com>]

Hi,

I am working on a project that is using Storm Trident for streaming
processing and I am
using opaque transaction trident Kafka spout to read data from Kafka,
process it and
save result to database.

The system is running for a while and we have more than one topologies
running. Now
we decided to slowly move our applications to new data center, we will have
new Kafka,
ZooKeeper, Storm and database cluster on new data center while still remain
the old
Kafka, ZooKeeper, Storm and database cluster for non-migrated topologies to
use.

I wonder if any of you have done this before and could you please share
your experience?
How do you migrate transactional state that Storm keeps in ZK? We can copy
snapshot
files and recover from there when we migrate the first topology but we
can't do it when
we migrate other topologies since at that time the two ZK cluster are
diverse.

Thanks
-Binh

Re: Move Storm to new data center

[Binh Nguyen Van <bi...@gmail.com>]

Thank you for the suggestion.

In our situation we can stop all the Kafka producers before migration so
the migration seems to be simpler.
The way that we came up is:

   - Stop all Kafka producers
   - Wait for Storm Trident to finish consuming remain messages
   - Shutdown Storm topology
   - Install Kafka, Storm, Zookeeper on new data center
   - Manually create currtx znode of Storm's coordinator and set its value
   to the value that we have in old data center
   - Deploy Storm topology to new data center and start consuming from the
   end of topic

I tested and it seems to work.

On Tue, Aug 11, 2015 at 11:59 PM, Kishore Senji <ks...@gmail.com> wrote:

> I haven't tried it myself, but I guess you would just need to sync the
> sub-tree for the topology that you are migrating from /transactional/<id>
> on the old cluster to the new cluster. /transactional is the default value
> of transactional.zookeeper.root unless you have modified.
>
> Similar is the case for KafkaSpout (non-trident) where the data is stored
> in /<root>/<spout-id>/
>
> On Tue, Aug 11, 2015 at 5:50 PM Binh Nguyen Van <bi...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am working on a project that is using Storm Trident for streaming
>> processing and I am
>> using opaque transaction trident Kafka spout to read data from Kafka,
>> process it and
>> save result to database.
>>
>> The system is running for a while and we have more than one topologies
>> running. Now
>> we decided to slowly move our applications to new data center, we will
>> have new Kafka,
>> ZooKeeper, Storm and database cluster on new data center while still
>> remain the old
>> Kafka, ZooKeeper, Storm and database cluster for non-migrated topologies
>> to use.
>>
>> I wonder if any of you have done this before and could you please share
>> your experience?
>> How do you migrate transactional state that Storm keeps in ZK? We can
>> copy snapshot
>> files and recover from there when we migrate the first topology but we
>> can't do it when
>> we migrate other topologies since at that time the two ZK cluster are
>> diverse.
>>
>> Thanks
>> -Binh
>>
>

Re: Move Storm to new data center

[Kishore Senji <ks...@gmail.com>]

I haven't tried it myself, but I guess you would just need to sync the
sub-tree for the topology that you are migrating from /transactional/<id>
on the old cluster to the new cluster. /transactional is the default value
of transactional.zookeeper.root unless you have modified.

Similar is the case for KafkaSpout (non-trident) where the data is stored
in /<root>/<spout-id>/

On Tue, Aug 11, 2015 at 5:50 PM Binh Nguyen Van <bi...@gmail.com> wrote:

> Hi,
>
> I am working on a project that is using Storm Trident for streaming
> processing and I am
> using opaque transaction trident Kafka spout to read data from Kafka,
> process it and
> save result to database.
>
> The system is running for a while and we have more than one topologies
> running. Now
> we decided to slowly move our applications to new data center, we will
> have new Kafka,
> ZooKeeper, Storm and database cluster on new data center while still
> remain the old
> Kafka, ZooKeeper, Storm and database cluster for non-migrated topologies
> to use.
>
> I wonder if any of you have done this before and could you please share
> your experience?
> How do you migrate transactional state that Storm keeps in ZK? We can copy
> snapshot
> files and recover from there when we migrate the first topology but we
> can't do it when
> we migrate other topologies since at that time the two ZK cluster are
> diverse.
>
> Thanks
> -Binh
>