You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Eric Stevens <mi...@gmail.com> on 2017/01/05 19:58:15 UTC
How does a node know it's fully joined to the ring?
I'm working on a bug report in my Deleting Compaction Strategy project.
Some of the provided deletion strategies allow for a series of deletion
rules to be added to a CQL table, before compaction begins, a snapshot of
the current rules are read out of the CQL table and a deletion plan is
produced that applies for the life of that compaction.
The bug report (at https://github.com/protectwise/cassandra-util/issues/2)
is that if compaction begins while a node is first starting up, before it
has joined the ring, the attempt to read the rules will fail with an
AssertionError (it's an attempt to call QueryProcessor.process at local
quorum before the node has joined the ring).
My question is, how can I detect when a node has fully joined the ring, and
it's safe to call QueryProcessor.process without just trying, failing, and
catching (and even then how would I know that it's because the node is
still starting up vs some other source of problems)? I had thought that
something might set a flag during the startup process to indicate that the
node thinks it is fully started up, but as far as I see nothing like that
is tracked.
I took a stab at what this could look like here:
https://github.com/protectwise/cassandra-util/blob/93dfb02157446e2e37427f1d4846c6039cc97099/deleting-compaction-strategy/src/main/java/com/protectwise/cassandra/retrospect/deletion/QueryHelper.java#L31-L36
This approach seems cumbersome and fragile, and I feel like I'm missing
something. How else could I tell whether it's safe to call out to
QueryProcessor.process yet?
Thanks in advance,
-e
Re: How does a node know it's fully joined to the ring?
Posted by Eric Stevens <mi...@gmail.com>.
Perfect, thanks!
On Thu, Jan 5, 2017 at 2:26 PM Brandon Williams <dr...@gmail.com> wrote:
> We don't bind thrift/CQL until bootstrap is complete, so that's not
> possible. But, if you want to be sure, you can check
> StorageService.isRPCServerRunning and isNativeTransportRunning.
>
> On Thu, Jan 5, 2017 at 3:17 PM, Eric Stevens <mi...@gmail.com> wrote:
>
> > Thanks for the quick response.
> >
> > Looking at it, I'm not sure if it's quite the right thing for me. Maybe
> I
> > used the wrong terminology; it actually doesn't matter if the node is
> still
> > bootstrapping, as long as it could query other nodes in the cluster.
> >
> > On Thu, Jan 5, 2017 at 1:03 PM Brandon Williams <dr...@gmail.com>
> wrote:
> >
> > > Probably want to do something like SystemKeyspace.bootstrapComplete()
> > >
> > > On Thu, Jan 5, 2017 at 1:58 PM, Eric Stevens <mi...@gmail.com>
> wrote:
> > >
> > > > I'm working on a bug report in my Deleting Compaction Strategy
> project.
> > > >
> > > > Some of the provided deletion strategies allow for a series of
> deletion
> > > > rules to be added to a CQL table, before compaction begins, a
> snapshot
> > of
> > > > the current rules are read out of the CQL table and a deletion plan
> is
> > > > produced that applies for the life of that compaction.
> > > >
> > > > The bug report (at
> > > https://github.com/protectwise/cassandra-util/issues/2)
> > > > is that if compaction begins while a node is first starting up,
> before
> > it
> > > > has joined the ring, the attempt to read the rules will fail with an
> > > > AssertionError (it's an attempt to call QueryProcessor.process at
> local
> > > > quorum before the node has joined the ring).
> > > >
> > > > My question is, how can I detect when a node has fully joined the
> ring,
> > > and
> > > > it's safe to call QueryProcessor.process without just trying,
> failing,
> > > and
> > > > catching (and even then how would I know that it's because the node
> is
> > > > still starting up vs some other source of problems)? I had thought
> > that
> > > > something might set a flag during the startup process to indicate
> that
> > > the
> > > > node thinks it is fully started up, but as far as I see nothing like
> > that
> > > > is tracked.
> > > >
> > > > I took a stab at what this could look like here:
> > > > https://github.com/protectwise/cassandra-util/blob/
> > > > 93dfb02157446e2e37427f1d4846c6039cc97099/deleting-
> > > > compaction-strategy/src/main/java/com/protectwise/
> > > > cassandra/retrospect/deletion/QueryHelper.java#L31-L36
> > > >
> > > > This approach seems cumbersome and fragile, and I feel like I'm
> missing
> > > > something. How else could I tell whether it's safe to call out to
> > > > QueryProcessor.process yet?
> > > >
> > > > Thanks in advance,
> > > > -e
> > > >
> > >
> >
>
Re: How does a node know it's fully joined to the ring?
Posted by Brandon Williams <dr...@gmail.com>.
We don't bind thrift/CQL until bootstrap is complete, so that's not
possible. But, if you want to be sure, you can check
StorageService.isRPCServerRunning and isNativeTransportRunning.
On Thu, Jan 5, 2017 at 3:17 PM, Eric Stevens <mi...@gmail.com> wrote:
> Thanks for the quick response.
>
> Looking at it, I'm not sure if it's quite the right thing for me. Maybe I
> used the wrong terminology; it actually doesn't matter if the node is still
> bootstrapping, as long as it could query other nodes in the cluster.
>
> On Thu, Jan 5, 2017 at 1:03 PM Brandon Williams <dr...@gmail.com> wrote:
>
> > Probably want to do something like SystemKeyspace.bootstrapComplete()
> >
> > On Thu, Jan 5, 2017 at 1:58 PM, Eric Stevens <mi...@gmail.com> wrote:
> >
> > > I'm working on a bug report in my Deleting Compaction Strategy project.
> > >
> > > Some of the provided deletion strategies allow for a series of deletion
> > > rules to be added to a CQL table, before compaction begins, a snapshot
> of
> > > the current rules are read out of the CQL table and a deletion plan is
> > > produced that applies for the life of that compaction.
> > >
> > > The bug report (at
> > https://github.com/protectwise/cassandra-util/issues/2)
> > > is that if compaction begins while a node is first starting up, before
> it
> > > has joined the ring, the attempt to read the rules will fail with an
> > > AssertionError (it's an attempt to call QueryProcessor.process at local
> > > quorum before the node has joined the ring).
> > >
> > > My question is, how can I detect when a node has fully joined the ring,
> > and
> > > it's safe to call QueryProcessor.process without just trying, failing,
> > and
> > > catching (and even then how would I know that it's because the node is
> > > still starting up vs some other source of problems)? I had thought
> that
> > > something might set a flag during the startup process to indicate that
> > the
> > > node thinks it is fully started up, but as far as I see nothing like
> that
> > > is tracked.
> > >
> > > I took a stab at what this could look like here:
> > > https://github.com/protectwise/cassandra-util/blob/
> > > 93dfb02157446e2e37427f1d4846c6039cc97099/deleting-
> > > compaction-strategy/src/main/java/com/protectwise/
> > > cassandra/retrospect/deletion/QueryHelper.java#L31-L36
> > >
> > > This approach seems cumbersome and fragile, and I feel like I'm missing
> > > something. How else could I tell whether it's safe to call out to
> > > QueryProcessor.process yet?
> > >
> > > Thanks in advance,
> > > -e
> > >
> >
>
Re: How does a node know it's fully joined to the ring?
Posted by Eric Stevens <mi...@gmail.com>.
Thanks for the quick response.
Looking at it, I'm not sure if it's quite the right thing for me. Maybe I
used the wrong terminology; it actually doesn't matter if the node is still
bootstrapping, as long as it could query other nodes in the cluster.
On Thu, Jan 5, 2017 at 1:03 PM Brandon Williams <dr...@gmail.com> wrote:
> Probably want to do something like SystemKeyspace.bootstrapComplete()
>
> On Thu, Jan 5, 2017 at 1:58 PM, Eric Stevens <mi...@gmail.com> wrote:
>
> > I'm working on a bug report in my Deleting Compaction Strategy project.
> >
> > Some of the provided deletion strategies allow for a series of deletion
> > rules to be added to a CQL table, before compaction begins, a snapshot of
> > the current rules are read out of the CQL table and a deletion plan is
> > produced that applies for the life of that compaction.
> >
> > The bug report (at
> https://github.com/protectwise/cassandra-util/issues/2)
> > is that if compaction begins while a node is first starting up, before it
> > has joined the ring, the attempt to read the rules will fail with an
> > AssertionError (it's an attempt to call QueryProcessor.process at local
> > quorum before the node has joined the ring).
> >
> > My question is, how can I detect when a node has fully joined the ring,
> and
> > it's safe to call QueryProcessor.process without just trying, failing,
> and
> > catching (and even then how would I know that it's because the node is
> > still starting up vs some other source of problems)? I had thought that
> > something might set a flag during the startup process to indicate that
> the
> > node thinks it is fully started up, but as far as I see nothing like that
> > is tracked.
> >
> > I took a stab at what this could look like here:
> > https://github.com/protectwise/cassandra-util/blob/
> > 93dfb02157446e2e37427f1d4846c6039cc97099/deleting-
> > compaction-strategy/src/main/java/com/protectwise/
> > cassandra/retrospect/deletion/QueryHelper.java#L31-L36
> >
> > This approach seems cumbersome and fragile, and I feel like I'm missing
> > something. How else could I tell whether it's safe to call out to
> > QueryProcessor.process yet?
> >
> > Thanks in advance,
> > -e
> >
>
Re: How does a node know it's fully joined to the ring?
Posted by Brandon Williams <dr...@gmail.com>.
Probably want to do something like SystemKeyspace.bootstrapComplete()
On Thu, Jan 5, 2017 at 1:58 PM, Eric Stevens <mi...@gmail.com> wrote:
> I'm working on a bug report in my Deleting Compaction Strategy project.
>
> Some of the provided deletion strategies allow for a series of deletion
> rules to be added to a CQL table, before compaction begins, a snapshot of
> the current rules are read out of the CQL table and a deletion plan is
> produced that applies for the life of that compaction.
>
> The bug report (at https://github.com/protectwise/cassandra-util/issues/2)
> is that if compaction begins while a node is first starting up, before it
> has joined the ring, the attempt to read the rules will fail with an
> AssertionError (it's an attempt to call QueryProcessor.process at local
> quorum before the node has joined the ring).
>
> My question is, how can I detect when a node has fully joined the ring, and
> it's safe to call QueryProcessor.process without just trying, failing, and
> catching (and even then how would I know that it's because the node is
> still starting up vs some other source of problems)? I had thought that
> something might set a flag during the startup process to indicate that the
> node thinks it is fully started up, but as far as I see nothing like that
> is tracked.
>
> I took a stab at what this could look like here:
> https://github.com/protectwise/cassandra-util/blob/
> 93dfb02157446e2e37427f1d4846c6039cc97099/deleting-
> compaction-strategy/src/main/java/com/protectwise/
> cassandra/retrospect/deletion/QueryHelper.java#L31-L36
>
> This approach seems cumbersome and fragile, and I feel like I'm missing
> something. How else could I tell whether it's safe to call out to
> QueryProcessor.process yet?
>
> Thanks in advance,
> -e
>