You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Guillermo Winkler <gw...@inconcertcc.com> on 2011/07/23 22:46:42 UTC
After column deletion cassandra won't insert more data to a specific key
I'm having a strange behavior on one of my cassandra boxes, after all
columns are removed from a row, insertion on that key stops working (from
API and from the cli)
[default@Agent] get Schedulers['atendimento'];
Returned 0 results.
[default@Agent] set Schedulers['atendimento']['test'] = 'dd';
Value inserted.
[default@Agent] get Schedulers['atendimento'];
Returned 0 results.
Already tried nodetool flush/compact/repair on the CF, doesn't fix the
problem.
With a ver simple setup:
* only one node in the cluster (the cluster never had more nodes nor
replicas)
* random partitioner
* CF defined as "create column family Schedulers with comparator=BytesType;"
The only way for it to start working again is to truncate the CF.
Do you have any clues how to diagnose what's going on?
Thanks,
Guille
Re: After column deletion cassandra won't insert more data to a
specific key
Posted by Guillermo Winkler <gw...@inconcertcc.com>.
I guess the problem it's not whether you can control time in a distributed
system or not, but in this case at least, it's if you consider a timestamp
set by a client outside the cluster as *safe*.
When the timestamp gets hidden behind a client/wrapper library
implementation default, realizing it's your responsibility to handle time
sync gets lost in the abstraction. Maybe it would be better for the cluster
to set a default value in those cases.
Not happening to me again, that's for sure :)
Thanks,
Guille
On Mon, Jul 25, 2011 at 7:48 PM, aaron morton <aa...@thelastpickle.com>wrote:
> It's just not possible to control time, as many super villains and Peter
> Schuller have shown us
> http://www.mail-archive.com/user@cassandra.apache.org/msg15636.html
>
> Often it's not necessary, you can design around simultaneous updates the
> same key, use a coordination layer such as zoo keeper or rely on consensus.
>
> If you have a design problem provide some details and someone may be able
> to help.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 26 Jul 2011, at 04:25, Guillermo Winkler wrote:
>
> Hi, thanks both for the answers.
>
> The problem was indeed with the timestamps.
>
> What was happening also was that in a mutation involving 1 deletion and
> various insertions for the same key, all were using the same timestamp, so
> beside looking at the code doing this
>
> remove key
> insert key, col, val
> insert key, col, val
> insert key, col, val
>
> With quorum 1, the insertions were always missing.
>
> I've been reading past threads regarding time sync inside/outside
> cassandra, I guess this ain't changing in the near future?
>
> Best,
> Guille
>
>
> On Sun, Jul 24, 2011 at 1:07 PM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> Remember the cli uses microsecond precision . so if your app is not using
>> the same precision weird this will result in clients writing the biggest
>> timsetamp winning the final value.
>>
>>
>> On Saturday, July 23, 2011, Jonathan Ellis <jb...@gmail.com> wrote:
>> > You must have given it a delete timestamp in the "future."
>> >
>> > On Sat, Jul 23, 2011 at 3:46 PM, Guillermo Winkler
>> > <gw...@inconcertcc.com> wrote:
>> >> I'm having a strange behavior on one of my cassandra boxes, after all
>> >> columns are removed from a row, insertion on that key stops working
>> (from
>> >> API and from the cli)
>> >> [default@Agent] get Schedulers['atendimento'];
>> >> Returned 0 results.
>> >> [default@Agent] set Schedulers['atendimento']['test'] = 'dd';
>> >> Value inserted.
>> >> [default@Agent] get Schedulers['atendimento'];
>> >> Returned 0 results.
>> >> Already tried nodetool flush/compact/repair on the CF, doesn't fix the
>> >> problem.
>> >> With a ver simple setup:
>> >> * only one node in the cluster (the cluster never had more nodes nor
>> >> replicas)
>> >> * random partitioner
>> >> * CF defined as "create column family Schedulers with
>> comparator=BytesType;"
>> >> The only way for it to start working again is to truncate the CF.
>> >> Do you have any clues how to diagnose what's going on?
>> >> Thanks,
>> >> Guille
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Jonathan Ellis
>> > Project Chair, Apache Cassandra
>> > co-founder of DataStax, the source for professional Cassandra support
>> > http://www.datastax.com
>> >
>>
>
>
>
Re: After column deletion cassandra won't insert more data to a specific key
Posted by aaron morton <aa...@thelastpickle.com>.
It's just not possible to control time, as many super villains and Peter Schuller have shown us http://www.mail-archive.com/user@cassandra.apache.org/msg15636.html
Often it's not necessary, you can design around simultaneous updates the same key, use a coordination layer such as zoo keeper or rely on consensus.
If you have a design problem provide some details and someone may be able to help.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 26 Jul 2011, at 04:25, Guillermo Winkler wrote:
> Hi, thanks both for the answers.
>
> The problem was indeed with the timestamps.
>
> What was happening also was that in a mutation involving 1 deletion and various insertions for the same key, all were using the same timestamp, so beside looking at the code doing this
>
> remove key
> insert key, col, val
> insert key, col, val
> insert key, col, val
>
> With quorum 1, the insertions were always missing.
>
> I've been reading past threads regarding time sync inside/outside cassandra, I guess this ain't changing in the near future?
>
> Best,
> Guille
>
>
> On Sun, Jul 24, 2011 at 1:07 PM, Edward Capriolo <ed...@gmail.com> wrote:
> Remember the cli uses microsecond precision . so if your app is not using the same precision weird this will result in clients writing the biggest timsetamp winning the final value.
>
>
> On Saturday, July 23, 2011, Jonathan Ellis <jb...@gmail.com> wrote:
> > You must have given it a delete timestamp in the "future."
> >
> > On Sat, Jul 23, 2011 at 3:46 PM, Guillermo Winkler
> > <gw...@inconcertcc.com> wrote:
> >> I'm having a strange behavior on one of my cassandra boxes, after all
> >> columns are removed from a row, insertion on that key stops working (from
> >> API and from the cli)
> >> [default@Agent] get Schedulers['atendimento'];
> >> Returned 0 results.
> >> [default@Agent] set Schedulers['atendimento']['test'] = 'dd';
> >> Value inserted.
> >> [default@Agent] get Schedulers['atendimento'];
> >> Returned 0 results.
> >> Already tried nodetool flush/compact/repair on the CF, doesn't fix the
> >> problem.
> >> With a ver simple setup:
> >> * only one node in the cluster (the cluster never had more nodes nor
> >> replicas)
> >> * random partitioner
> >> * CF defined as "create column family Schedulers with comparator=BytesType;"
> >> The only way for it to start working again is to truncate the CF.
> >> Do you have any clues how to diagnose what's going on?
> >> Thanks,
> >> Guille
> >>
> >>
> >
> >
> >
> > --
> > Jonathan Ellis
> > Project Chair, Apache Cassandra
> > co-founder of DataStax, the source for professional Cassandra support
> > http://www.datastax.com
> >
>
Re: After column deletion cassandra won't insert more data to a
specific key
Posted by Guillermo Winkler <gw...@inconcertcc.com>.
Hi, thanks both for the answers.
The problem was indeed with the timestamps.
What was happening also was that in a mutation involving 1 deletion and
various insertions for the same key, all were using the same timestamp, so
beside looking at the code doing this
remove key
insert key, col, val
insert key, col, val
insert key, col, val
With quorum 1, the insertions were always missing.
I've been reading past threads regarding time sync inside/outside cassandra,
I guess this ain't changing in the near future?
Best,
Guille
On Sun, Jul 24, 2011 at 1:07 PM, Edward Capriolo <ed...@gmail.com>wrote:
> Remember the cli uses microsecond precision . so if your app is not using
> the same precision weird this will result in clients writing the biggest
> timsetamp winning the final value.
>
>
> On Saturday, July 23, 2011, Jonathan Ellis <jb...@gmail.com> wrote:
> > You must have given it a delete timestamp in the "future."
> >
> > On Sat, Jul 23, 2011 at 3:46 PM, Guillermo Winkler
> > <gw...@inconcertcc.com> wrote:
> >> I'm having a strange behavior on one of my cassandra boxes, after all
> >> columns are removed from a row, insertion on that key stops working
> (from
> >> API and from the cli)
> >> [default@Agent] get Schedulers['atendimento'];
> >> Returned 0 results.
> >> [default@Agent] set Schedulers['atendimento']['test'] = 'dd';
> >> Value inserted.
> >> [default@Agent] get Schedulers['atendimento'];
> >> Returned 0 results.
> >> Already tried nodetool flush/compact/repair on the CF, doesn't fix the
> >> problem.
> >> With a ver simple setup:
> >> * only one node in the cluster (the cluster never had more nodes nor
> >> replicas)
> >> * random partitioner
> >> * CF defined as "create column family Schedulers with
> comparator=BytesType;"
> >> The only way for it to start working again is to truncate the CF.
> >> Do you have any clues how to diagnose what's going on?
> >> Thanks,
> >> Guille
> >>
> >>
> >
> >
> >
> > --
> > Jonathan Ellis
> > Project Chair, Apache Cassandra
> > co-founder of DataStax, the source for professional Cassandra support
> > http://www.datastax.com
> >
>
Re: After column deletion cassandra won't insert more data to a
specific key
Posted by Edward Capriolo <ed...@gmail.com>.
Remember the cli uses microsecond precision . so if your app is not using
the same precision weird this will result in clients writing the biggest
timsetamp winning the final value.
On Saturday, July 23, 2011, Jonathan Ellis <jb...@gmail.com> wrote:
> You must have given it a delete timestamp in the "future."
>
> On Sat, Jul 23, 2011 at 3:46 PM, Guillermo Winkler
> <gw...@inconcertcc.com> wrote:
>> I'm having a strange behavior on one of my cassandra boxes, after all
>> columns are removed from a row, insertion on that key stops working (from
>> API and from the cli)
>> [default@Agent] get Schedulers['atendimento'];
>> Returned 0 results.
>> [default@Agent] set Schedulers['atendimento']['test'] = 'dd';
>> Value inserted.
>> [default@Agent] get Schedulers['atendimento'];
>> Returned 0 results.
>> Already tried nodetool flush/compact/repair on the CF, doesn't fix the
>> problem.
>> With a ver simple setup:
>> * only one node in the cluster (the cluster never had more nodes nor
>> replicas)
>> * random partitioner
>> * CF defined as "create column family Schedulers with
comparator=BytesType;"
>> The only way for it to start working again is to truncate the CF.
>> Do you have any clues how to diagnose what's going on?
>> Thanks,
>> Guille
>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: After column deletion cassandra won't insert more data to a
specific key
Posted by Jonathan Ellis <jb...@gmail.com>.
You must have given it a delete timestamp in the "future."
On Sat, Jul 23, 2011 at 3:46 PM, Guillermo Winkler
<gw...@inconcertcc.com> wrote:
> I'm having a strange behavior on one of my cassandra boxes, after all
> columns are removed from a row, insertion on that key stops working (from
> API and from the cli)
> [default@Agent] get Schedulers['atendimento'];
> Returned 0 results.
> [default@Agent] set Schedulers['atendimento']['test'] = 'dd';
> Value inserted.
> [default@Agent] get Schedulers['atendimento'];
> Returned 0 results.
> Already tried nodetool flush/compact/repair on the CF, doesn't fix the
> problem.
> With a ver simple setup:
> * only one node in the cluster (the cluster never had more nodes nor
> replicas)
> * random partitioner
> * CF defined as "create column family Schedulers with comparator=BytesType;"
> The only way for it to start working again is to truncate the CF.
> Do you have any clues how to diagnose what's going on?
> Thanks,
> Guille
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: After column deletion cassandra won't insert more data to a
specific key
Posted by Guillermo Winkler <gw...@inconcertcc.com>.
Sorry, Cassandra version is 0.7.4
On Sat, Jul 23, 2011 at 5:46 PM, Guillermo Winkler <gwinkler@inconcertcc.com
> wrote:
> I'm having a strange behavior on one of my cassandra boxes, after all
> columns are removed from a row, insertion on that key stops working (from
> API and from the cli)
>
> [default@Agent] get Schedulers['atendimento'];
> Returned 0 results.
> [default@Agent] set Schedulers['atendimento']['test'] = 'dd';
> Value inserted.
> [default@Agent] get Schedulers['atendimento'];
> Returned 0 results.
>
> Already tried nodetool flush/compact/repair on the CF, doesn't fix the
> problem.
>
> With a ver simple setup:
> * only one node in the cluster (the cluster never had more nodes nor
> replicas)
> * random partitioner
> * CF defined as "create column family Schedulers with
> comparator=BytesType;"
>
> The only way for it to start working again is to truncate the CF.
>
> Do you have any clues how to diagnose what's going on?
>
> Thanks,
> Guille
>
>
>