You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Vitalii Tymchyshyn <vi...@tym.im> on 2015/02/04 21:10:49 UTC
Implementing sharding with zookeeper
Hello.
I have a task that looks like a fit for Zookeeper and I thought I find a
recipe in Curator, but I dont. So, It looks like I need my own recipe.
The problem is: we have a stream of tasks coming in. All tasks for the same
account must be done sequential. But for different accounts it's OK to do
in parallel, so we want workers to distribute account space between them.
The first step, of course, is to make an account hash to ensure load is
distributed as smooth as possible.
Next I could simply use a lot of locks, but this way I can't ensure
fairness.
Thats why I think I need a Leader who will distribute (or redistribute in
case of node death) the hash space between the live nodes.
What do you think? May be I am missing something and there is existing
recipe for the task?
Best regards, Vitalii Tymchyshyn
Re: Implementing sharding with zookeeper
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
I wrote something that might work for you. Have a look at Nirmata Workflow: http://nirmataoss.github.io/workflow
====================
Jordan Zimmerman
> On Feb 4, 2015, at 12:10 PM, Vitalii Tymchyshyn <vi...@tym.im> wrote:
>
> Hello.
>
> I have a task that looks like a fit for Zookeeper and I thought I find a
> recipe in Curator, but I dont. So, It looks like I need my own recipe.
>
> The problem is: we have a stream of tasks coming in. All tasks for the same
> account must be done sequential. But for different accounts it's OK to do
> in parallel, so we want workers to distribute account space between them.
>
> The first step, of course, is to make an account hash to ensure load is
> distributed as smooth as possible.
>
> Next I could simply use a lot of locks, but this way I can't ensure
> fairness.
>
> Thats why I think I need a Leader who will distribute (or redistribute in
> case of node death) the hash space between the live nodes.
>
> What do you think? May be I am missing something and there is existing
> recipe for the task?
>
> Best regards, Vitalii Tymchyshyn
Re: Implementing sharding with zookeeper
Posted by Vitalii Tymchyshyn <vi...@tym.im>.
Thanks, it looks promising.
Best regards, Vitalii Tymchyshyn
Ср, 4 лют. 2015 15:20 kishore g <g....@gmail.com> пише:
> You might have to look at Apache helix. See this recipe
> http://helix.apache.org/0.6.2-incubating-docs/recipes/lock_manager.html
>
> On Wed, Feb 4, 2015 at 12:10 PM, Vitalii Tymchyshyn <vi...@tym.im> wrote:
>
> > Hello.
> >
> > I have a task that looks like a fit for Zookeeper and I thought I find a
> > recipe in Curator, but I dont. So, It looks like I need my own recipe.
> >
> > The problem is: we have a stream of tasks coming in. All tasks for the
> same
> > account must be done sequential. But for different accounts it's OK to do
> > in parallel, so we want workers to distribute account space between them.
> >
> > The first step, of course, is to make an account hash to ensure load is
> > distributed as smooth as possible.
> >
> > Next I could simply use a lot of locks, but this way I can't ensure
> > fairness.
> >
> > Thats why I think I need a Leader who will distribute (or redistribute in
> > case of node death) the hash space between the live nodes.
> >
> > What do you think? May be I am missing something and there is existing
> > recipe for the task?
> >
> > Best regards, Vitalii Tymchyshyn
> >
>
Re: Implementing sharding with zookeeper
Posted by kishore g <g....@gmail.com>.
You might have to look at Apache helix. See this recipe
http://helix.apache.org/0.6.2-incubating-docs/recipes/lock_manager.html
On Wed, Feb 4, 2015 at 12:10 PM, Vitalii Tymchyshyn <vi...@tym.im> wrote:
> Hello.
>
> I have a task that looks like a fit for Zookeeper and I thought I find a
> recipe in Curator, but I dont. So, It looks like I need my own recipe.
>
> The problem is: we have a stream of tasks coming in. All tasks for the same
> account must be done sequential. But for different accounts it's OK to do
> in parallel, so we want workers to distribute account space between them.
>
> The first step, of course, is to make an account hash to ensure load is
> distributed as smooth as possible.
>
> Next I could simply use a lot of locks, but this way I can't ensure
> fairness.
>
> Thats why I think I need a Leader who will distribute (or redistribute in
> case of node death) the hash space between the live nodes.
>
> What do you think? May be I am missing something and there is existing
> recipe for the task?
>
> Best regards, Vitalii Tymchyshyn
>