Schema disagreement under normal conditions, ALTER TABLE hangs

[Josh Dzielak <jo...@keen.io>]

Recently we had a strange thing happen. Altering schema (gc_grace_seconds) for a column family resulted in a schema disagreement. 3/4 of nodes got it, 1/4 didn't. There was no partition at the time, nor was there multiple schema updates issued. Going to the nodes with stale schema and trying to do the ALTER TABLE there resulted in hanging. We were eventually able to get schema agreement by restarting nodes, but both the initial disagreement under normal conditions and the hanging ALTER TABLE seem pretty weird. Any ideas here? Sound like a bug?  

We're on 1.2.8.

Thanks,
Josh

--
Josh Dzielak • Keen IO • @dzello (https://twitter.com/dzello)

Re: Schema disagreement under normal conditions, ALTER TABLE hangs

[Josh Dzielak <jo...@keen.io>]

Thanks Rob. Let me add one thing in case someone else finds this thread - 

Restarting the nodes did not in and of itself get the schema disagreement resolved. We had to run the ALTER TABLE command individually on each of the disagreeing nodes once they came back up. 

On Tuesday, November 26, 2013 at 11:24 AM, Robert Coli wrote:

> On Mon, Nov 25, 2013 at 6:42 PM, Josh Dzielak <josh@keen.io (mailto:josh@keen.io)> wrote:
> > Recently we had a strange thing happen. Altering schema (gc_grace_seconds) for a column family resulted in a schema disagreement. 3/4 of nodes got it, 1/4 didn't. There was no partition at the time, nor was there multiple schema updates issued. Going to the nodes with stale schema and trying to do the ALTER TABLE there resulted in hanging. We were eventually able to get schema agreement by restarting nodes, but both the initial disagreement under normal conditions and the hanging ALTER TABLE seem pretty weird. Any ideas here? Sound like a bug? 
> 
> Yes, that sounds like a bug. This behavior is less common in 1.2.x than it was previously, but still happens sometimes. It's interesting that restarting the affected node helped, in previous versions of "hung schema" issue, it would survive restart. 
>  
> > We're on 1.2.8.
> > 
> 
> 
> Unfortunately, unless you have a repro path, it is probably not worth reporting a JIRA. 
> 
> =Rob
>  
> 
> 
> 
> 
> 

Re: Schema disagreement under normal conditions, ALTER TABLE hangs

[Robert Coli <rc...@eventbrite.com>]

On Mon, Nov 25, 2013 at 6:42 PM, Josh Dzielak <jo...@keen.io> wrote:

> Recently we had a strange thing happen. Altering schema (gc_grace_seconds)
> for a column family resulted in a schema disagreement. 3/4 of nodes got it,
> 1/4 didn't. There was no partition at the time, nor was there multiple
> schema updates issued. Going to the nodes with stale schema and trying to
> do the ALTER TABLE there resulted in hanging. We were eventually able to
> get schema agreement by restarting nodes, but both the initial disagreement
> under normal conditions and the hanging ALTER TABLE seem pretty weird. Any
> ideas here? Sound like a bug?
>

Yes, that sounds like a bug. This behavior is less common in 1.2.x than it
was previously, but still happens sometimes. It's interesting that
restarting the affected node helped, in previous versions of "hung schema"
issue, it would survive restart.


> We're on 1.2.8.
>

Unfortunately, unless you have a repro path, it is probably not worth
reporting a JIRA.

=Rob