You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Lucas Nodine <lu...@gmail.com> on 2010/09/07 16:11:23 UTC
batch_mutate silently failing in Cassandra
Hello all,
I have posted the following to Stackoverflow, but thought that I would also
try the list. If you have any suggestions, please let me know
I am working with Cassandra 0.6.5 using the thrift interface. I am trying to
use the batch_mutate method call, however, when I execute it, I receive no
error message. This leads me to believe it worked. When I check using the
CLI, there is nothing there. Is there something wrong with my code or format
of the mutation_map that anyone can see? Any ideas?
Thanks in advance,
LN
public void Update(string keyspace, Common.NetworkPackage.MetaAsset ma) {
Dictionary<string, Dictionary<string, List<Mutation>>> package;
Dictionary<string, List<Mutation>> packageEntry;
Dictionary<string, object>.Enumerator en;
List<Mutation> mutList;
Mutation mut;
DateTime now = DateTime.Now;
if(!ma.Fields.ContainsKey("$guid"))
throw new ArgumentException("The field $guid is not present");
mutList = new List<Mutation>();
en = ma.Fields.GetEnumerator();
while(en.MoveNext())
{
if (en.Current.Value == null)
continue;
mut = new Mutation();
mut.Column_or_supercolumn = new ColumnOrSuperColumn();
mut.Column_or_supercolumn.Column = new Column();
mut.Column_or_supercolumn.Column.Name = _utf8.GetBytes(en.Current.Key);
if (en.Current.Value == null)
mut.Column_or_supercolumn.Column.Value = null;
else
mut.Column_or_supercolumn.Column.Value = ToBytes(en.Current.Value);
mut.Column_or_supercolumn.Column.Timestamp =
Utilities.Timestamp(now);
mutList.Add(mut);
}
packageEntry = new Dictionary<string, List<Mutation>>();
packageEntry.Add("MetaAsset", mutList);
package = new Dictionary<string, Dictionary<string, List<Mutation>>>();
package.Add((string)ma.Fields["$guid"], packageEntry);
Console.WriteLine(Utilities.ExportBulkMutate("LawOffice", package));
_client.batch_mutate(keyspace, package, ConsistencyLevel.QUORUM); }
The above code produces (columns are name:value @ timestamp, value consists
of a type:and a representation of the actual value):
LawOffice : {
Row=08469fba50f448be8943614abd059d10 : {
CF=MetaAsset : {
$guid : String:08469fba50f448be8943614abd059d10 @ 93
$creator : String:Lucas @ 93
$previousversion : String:00000000000000000000000000000000 @ 93
$nextversion : String:00000000000000000000000000000000 @ 93
$etag : String:0 @ 93
$length : Int32:123456789 @ 93
$extension : String:.odt @ 93
$created : DateTime:90 @ 93
$modified : DateTime:90 @ 93
$lastaccess : DateTime:90 @ 93
$title : String:Title @ 93
$tags : List`1:tag1,tag2,tag3 @ 93
}
}
}
Re: batch_mutate silently failing in Cassandra
Posted by Jonathan Ellis <jb...@gmail.com>.
QUORUM of ReplicationFactor=1 is 1. All consistency levels should
work fine as long as RF <= node count. If you are seeing it work at
CL.ONE but not at QUORUM then that is probably a bug.
On Tue, Sep 7, 2010 at 7:36 PM, Lucas Nodine <lu...@gmail.com> wrote:
> Partially Resolved...
>
> Taking your advise (both Jonathan and Aaron's) I was able to track the
> problem down. The issue was that running insert (also batch_mutate) and
> using a consistencylevel of quorum on a cluster of 1 server with a
> replication factor of 1 does not work. Well, that is not accurate, let me
> explain. I believe the problem lies in the description of QUORUM as defined
> in the API on the wiki "Ensure that the write has been written to N / 2 + 1
> replicas before responding to the client."
>
> 1/2+1 = 1.5 -- One server can never accomplish this. I believe I read
> something about this somewhere in all the documentation, but it would be
> really handy to have received an error telling me that it was impossible
> (assuming this was actually the issue). I am by far, not familiar with the
> replication model, but if it is not by design, someone let me know and I'll
> report it.
>
> Thx,
>
> - LN
>
> On Tue, Sep 7, 2010 at 3:00 PM, Lucas Nodine <lu...@gmail.com> wrote:
>>
>> Jonathan,
>>
>> I have done it successfully with insert, but I have not tried it with
>> mutate. I'll give that a try tonight.
>>
>> Thanks
>> On Tue, Sep 7, 2010 at 2:54 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>
>>> I would try to get a single hard-coded column to insert, before doing
>>> something more complex.
>>>
>>> You can also enable debug logging on the server and see if that
>>> matches what you want the client to be doing.
>>>
>>> On Tue, Sep 7, 2010 at 9:11 AM, Lucas Nodine <lu...@gmail.com>
>>> wrote:
>>> > Hello all,
>>> >
>>> > I have posted the following to Stackoverflow, but thought that I would
>>> > also
>>> > try the list. If you have any suggestions, please let me know
>>> >
>>> > I am working with Cassandra 0.6.5 using the thrift interface. I am
>>> > trying to
>>> > use the batch_mutate method call, however, when I execute it, I receive
>>> > no
>>> > error message. This leads me to believe it worked. When I check using
>>> > the
>>> > CLI, there is nothing there. Is there something wrong with my code or
>>> > format
>>> > of the mutation_map that anyone can see? Any ideas?
>>> >
>>> > Thanks in advance,
>>> >
>>> > LN
>>> >
>>> > public void Update(string keyspace, Common.NetworkPackage.MetaAsset ma)
>>> > {
>>> > Dictionary<string, Dictionary<string, List<Mutation>>> package;
>>> > Dictionary<string, List<Mutation>> packageEntry;
>>> > Dictionary<string, object>.Enumerator en;
>>> > List<Mutation> mutList;
>>> > Mutation mut;
>>> > DateTime now = DateTime.Now;
>>> >
>>> > if(!ma.Fields.ContainsKey("$guid"))
>>> > throw new ArgumentException("The field $guid is not present");
>>> >
>>> > mutList = new List<Mutation>();
>>> > en = ma.Fields.GetEnumerator();
>>> >
>>> > while(en.MoveNext())
>>> > {
>>> > if (en.Current.Value == null)
>>> > continue;
>>> > mut = new Mutation();
>>> > mut.Column_or_supercolumn = new ColumnOrSuperColumn();
>>> > mut.Column_or_supercolumn.Column = new Column();
>>> > mut.Column_or_supercolumn.Column.Name =
>>> > _utf8.GetBytes(en.Current.Key);
>>> >
>>> > if (en.Current.Value == null)
>>> > mut.Column_or_supercolumn.Column.Value = null;
>>> > else
>>> > mut.Column_or_supercolumn.Column.Value =
>>> > ToBytes(en.Current.Value);
>>> >
>>> > mut.Column_or_supercolumn.Column.Timestamp =
>>> > Utilities.Timestamp(now);
>>> >
>>> > mutList.Add(mut);
>>> > }
>>> >
>>> > packageEntry = new Dictionary<string, List<Mutation>>();
>>> > packageEntry.Add("MetaAsset", mutList);
>>> >
>>> > package = new Dictionary<string, Dictionary<string,
>>> > List<Mutation>>>();
>>> > package.Add((string)ma.Fields["$guid"], packageEntry);
>>> >
>>> > Console.WriteLine(Utilities.ExportBulkMutate("LawOffice", package));
>>> >
>>> > _client.batch_mutate(keyspace, package, ConsistencyLevel.QUORUM);
>>> > }
>>> >
>>> > The above code produces (columns are name:value @ timestamp, value
>>> > consists
>>> > of a type:and a representation of the actual value):
>>> >
>>> > LawOffice : {
>>> > Row=08469fba50f448be8943614abd059d10 : {
>>> > CF=MetaAsset : {
>>> > $guid : String:08469fba50f448be8943614abd059d10 @ 93
>>> > $creator : String:Lucas @ 93
>>> > $previousversion : String:00000000000000000000000000000000 @ 93
>>> > $nextversion : String:00000000000000000000000000000000 @ 93
>>> > $etag : String:0 @ 93
>>> > $length : Int32:123456789 @ 93
>>> > $extension : String:.odt @ 93
>>> > $created : DateTime:90 @ 93
>>> > $modified : DateTime:90 @ 93
>>> > $lastaccess : DateTime:90 @ 93
>>> > $title : String:Title @ 93
>>> > $tags : List`1:tag1,tag2,tag3 @ 93
>>> > }
>>> > }
>>> > }
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of Riptano, the source for professional Cassandra support
>>> http://riptano.com
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: batch_mutate silently failing in Cassandra
Posted by Lucas Nodine <lu...@gmail.com>.
Partially Resolved...
Taking your advise (both Jonathan and Aaron's) I was able to track the
problem down. The issue was that running insert (also batch_mutate) and
using a consistencylevel of quorum on a cluster of 1 server with a
replication factor of 1 does not work. Well, that is not accurate, let me
explain. I believe the problem lies in the description of QUORUM as defined
in the API on the wiki "Ensure that the write has been written to N /
2 + 1replicas before responding to the client."
1/2+1 = 1.5 -- One server can never accomplish this. I believe I read
something about this somewhere in all the documentation, but it would be
really handy to have received an error telling me that it was impossible
(assuming this was actually the issue). I am by far, not familiar with the
replication model, but if it is not by design, someone let me know and I'll
report it.
Thx,
- LN
On Tue, Sep 7, 2010 at 3:00 PM, Lucas Nodine <lu...@gmail.com> wrote:
> Jonathan,
>
> I have done it successfully with insert, but I have not tried it with
> mutate. I'll give that a try tonight.
>
> Thanks
> On Tue, Sep 7, 2010 at 2:54 PM, Jonathan Ellis <jb...@gmail.com>wrote:
>
>> I would try to get a single hard-coded column to insert, before doing
>> something more complex.
>>
>> You can also enable debug logging on the server and see if that
>> matches what you want the client to be doing.
>>
>> On Tue, Sep 7, 2010 at 9:11 AM, Lucas Nodine <lu...@gmail.com>
>> wrote:
>> > Hello all,
>> >
>> > I have posted the following to Stackoverflow, but thought that I would
>> also
>> > try the list. If you have any suggestions, please let me know
>> >
>> > I am working with Cassandra 0.6.5 using the thrift interface. I am
>> trying to
>> > use the batch_mutate method call, however, when I execute it, I receive
>> no
>> > error message. This leads me to believe it worked. When I check using
>> the
>> > CLI, there is nothing there. Is there something wrong with my code or
>> format
>> > of the mutation_map that anyone can see? Any ideas?
>> >
>> > Thanks in advance,
>> >
>> > LN
>> >
>> > public void Update(string keyspace, Common.NetworkPackage.MetaAsset ma)
>> > {
>> > Dictionary<string, Dictionary<string, List<Mutation>>> package;
>> > Dictionary<string, List<Mutation>> packageEntry;
>> > Dictionary<string, object>.Enumerator en;
>> > List<Mutation> mutList;
>> > Mutation mut;
>> > DateTime now = DateTime.Now;
>> >
>> > if(!ma.Fields.ContainsKey("$guid"))
>> > throw new ArgumentException("The field $guid is not present");
>> >
>> > mutList = new List<Mutation>();
>> > en = ma.Fields.GetEnumerator();
>> >
>> > while(en.MoveNext())
>> > {
>> > if (en.Current.Value == null)
>> > continue;
>> > mut = new Mutation();
>> > mut.Column_or_supercolumn = new ColumnOrSuperColumn();
>> > mut.Column_or_supercolumn.Column = new Column();
>> > mut.Column_or_supercolumn.Column.Name<http://mut.column_or_supercolumn.column.name/>= _utf8.GetBytes(en.Current.Key);
>> >
>> > if (en.Current.Value == null)
>> > mut.Column_or_supercolumn.Column.Value = null;
>> > else
>> > mut.Column_or_supercolumn.Column.Value =
>> ToBytes(en.Current.Value);
>> >
>> > mut.Column_or_supercolumn.Column.Timestamp =
>> Utilities.Timestamp(now);
>> >
>> > mutList.Add(mut);
>> > }
>> >
>> > packageEntry = new Dictionary<string, List<Mutation>>();
>> > packageEntry.Add("MetaAsset", mutList);
>> >
>> > package = new Dictionary<string, Dictionary<string,
>> List<Mutation>>>();
>> > package.Add((string)ma.Fields["$guid"], packageEntry);
>> >
>> > Console.WriteLine(Utilities.ExportBulkMutate("LawOffice", package));
>> >
>> > _client.batch_mutate(keyspace, package, ConsistencyLevel.QUORUM);
>> > }
>> >
>> > The above code produces (columns are name:value @ timestamp, value
>> consists
>> > of a type:and a representation of the actual value):
>> >
>> > LawOffice : {
>> > Row=08469fba50f448be8943614abd059d10 : {
>> > CF=MetaAsset : {
>> > $guid : String:08469fba50f448be8943614abd059d10 @ 93
>> > $creator : String:Lucas @ 93
>> > $previousversion : String:00000000000000000000000000000000 @ 93
>> > $nextversion : String:00000000000000000000000000000000 @ 93
>> > $etag : String:0 @ 93
>> > $length : Int32:123456789 @ 93
>> > $extension : String:.odt @ 93
>> > $created : DateTime:90 @ 93
>> > $modified : DateTime:90 @ 93
>> > $lastaccess : DateTime:90 @ 93
>> > $title : String:Title @ 93
>> > $tags : List`1:tag1,tag2,tag3 @ 93
>> > }
>> > }
>> > }
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>>
>
Re: batch_mutate silently failing in Cassandra
Posted by Lucas Nodine <lu...@gmail.com>.
Jonathan,
I have done it successfully with insert, but I have not tried it with
mutate. I'll give that a try tonight.
Thanks
On Tue, Sep 7, 2010 at 2:54 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> I would try to get a single hard-coded column to insert, before doing
> something more complex.
>
> You can also enable debug logging on the server and see if that
> matches what you want the client to be doing.
>
> On Tue, Sep 7, 2010 at 9:11 AM, Lucas Nodine <lu...@gmail.com>
> wrote:
> > Hello all,
> >
> > I have posted the following to Stackoverflow, but thought that I would
> also
> > try the list. If you have any suggestions, please let me know
> >
> > I am working with Cassandra 0.6.5 using the thrift interface. I am trying
> to
> > use the batch_mutate method call, however, when I execute it, I receive
> no
> > error message. This leads me to believe it worked. When I check using the
> > CLI, there is nothing there. Is there something wrong with my code or
> format
> > of the mutation_map that anyone can see? Any ideas?
> >
> > Thanks in advance,
> >
> > LN
> >
> > public void Update(string keyspace, Common.NetworkPackage.MetaAsset ma)
> > {
> > Dictionary<string, Dictionary<string, List<Mutation>>> package;
> > Dictionary<string, List<Mutation>> packageEntry;
> > Dictionary<string, object>.Enumerator en;
> > List<Mutation> mutList;
> > Mutation mut;
> > DateTime now = DateTime.Now;
> >
> > if(!ma.Fields.ContainsKey("$guid"))
> > throw new ArgumentException("The field $guid is not present");
> >
> > mutList = new List<Mutation>();
> > en = ma.Fields.GetEnumerator();
> >
> > while(en.MoveNext())
> > {
> > if (en.Current.Value == null)
> > continue;
> > mut = new Mutation();
> > mut.Column_or_supercolumn = new ColumnOrSuperColumn();
> > mut.Column_or_supercolumn.Column = new Column();
> > mut.Column_or_supercolumn.Column.Name<http://mut.column_or_supercolumn.column.name/>= _utf8.GetBytes(en.Current.Key);
> >
> > if (en.Current.Value == null)
> > mut.Column_or_supercolumn.Column.Value = null;
> > else
> > mut.Column_or_supercolumn.Column.Value = ToBytes(en.Current.Value);
> >
> > mut.Column_or_supercolumn.Column.Timestamp =
> Utilities.Timestamp(now);
> >
> > mutList.Add(mut);
> > }
> >
> > packageEntry = new Dictionary<string, List<Mutation>>();
> > packageEntry.Add("MetaAsset", mutList);
> >
> > package = new Dictionary<string, Dictionary<string, List<Mutation>>>();
> > package.Add((string)ma.Fields["$guid"], packageEntry);
> >
> > Console.WriteLine(Utilities.ExportBulkMutate("LawOffice", package));
> >
> > _client.batch_mutate(keyspace, package, ConsistencyLevel.QUORUM);
> > }
> >
> > The above code produces (columns are name:value @ timestamp, value
> consists
> > of a type:and a representation of the actual value):
> >
> > LawOffice : {
> > Row=08469fba50f448be8943614abd059d10 : {
> > CF=MetaAsset : {
> > $guid : String:08469fba50f448be8943614abd059d10 @ 93
> > $creator : String:Lucas @ 93
> > $previousversion : String:00000000000000000000000000000000 @ 93
> > $nextversion : String:00000000000000000000000000000000 @ 93
> > $etag : String:0 @ 93
> > $length : Int32:123456789 @ 93
> > $extension : String:.odt @ 93
> > $created : DateTime:90 @ 93
> > $modified : DateTime:90 @ 93
> > $lastaccess : DateTime:90 @ 93
> > $title : String:Title @ 93
> > $tags : List`1:tag1,tag2,tag3 @ 93
> > }
> > }
> > }
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: batch_mutate silently failing in Cassandra
Posted by Jonathan Ellis <jb...@gmail.com>.
I would try to get a single hard-coded column to insert, before doing
something more complex.
You can also enable debug logging on the server and see if that
matches what you want the client to be doing.
On Tue, Sep 7, 2010 at 9:11 AM, Lucas Nodine <lu...@gmail.com> wrote:
> Hello all,
>
> I have posted the following to Stackoverflow, but thought that I would also
> try the list. If you have any suggestions, please let me know
>
> I am working with Cassandra 0.6.5 using the thrift interface. I am trying to
> use the batch_mutate method call, however, when I execute it, I receive no
> error message. This leads me to believe it worked. When I check using the
> CLI, there is nothing there. Is there something wrong with my code or format
> of the mutation_map that anyone can see? Any ideas?
>
> Thanks in advance,
>
> LN
>
> public void Update(string keyspace, Common.NetworkPackage.MetaAsset ma)
> {
> Dictionary<string, Dictionary<string, List<Mutation>>> package;
> Dictionary<string, List<Mutation>> packageEntry;
> Dictionary<string, object>.Enumerator en;
> List<Mutation> mutList;
> Mutation mut;
> DateTime now = DateTime.Now;
>
> if(!ma.Fields.ContainsKey("$guid"))
> throw new ArgumentException("The field $guid is not present");
>
> mutList = new List<Mutation>();
> en = ma.Fields.GetEnumerator();
>
> while(en.MoveNext())
> {
> if (en.Current.Value == null)
> continue;
> mut = new Mutation();
> mut.Column_or_supercolumn = new ColumnOrSuperColumn();
> mut.Column_or_supercolumn.Column = new Column();
> mut.Column_or_supercolumn.Column.Name = _utf8.GetBytes(en.Current.Key);
>
> if (en.Current.Value == null)
> mut.Column_or_supercolumn.Column.Value = null;
> else
> mut.Column_or_supercolumn.Column.Value = ToBytes(en.Current.Value);
>
> mut.Column_or_supercolumn.Column.Timestamp = Utilities.Timestamp(now);
>
> mutList.Add(mut);
> }
>
> packageEntry = new Dictionary<string, List<Mutation>>();
> packageEntry.Add("MetaAsset", mutList);
>
> package = new Dictionary<string, Dictionary<string, List<Mutation>>>();
> package.Add((string)ma.Fields["$guid"], packageEntry);
>
> Console.WriteLine(Utilities.ExportBulkMutate("LawOffice", package));
>
> _client.batch_mutate(keyspace, package, ConsistencyLevel.QUORUM);
> }
>
> The above code produces (columns are name:value @ timestamp, value consists
> of a type:and a representation of the actual value):
>
> LawOffice : {
> Row=08469fba50f448be8943614abd059d10 : {
> CF=MetaAsset : {
> $guid : String:08469fba50f448be8943614abd059d10 @ 93
> $creator : String:Lucas @ 93
> $previousversion : String:00000000000000000000000000000000 @ 93
> $nextversion : String:00000000000000000000000000000000 @ 93
> $etag : String:0 @ 93
> $length : Int32:123456789 @ 93
> $extension : String:.odt @ 93
> $created : DateTime:90 @ 93
> $modified : DateTime:90 @ 93
> $lastaccess : DateTime:90 @ 93
> $title : String:Title @ 93
> $tags : List`1:tag1,tag2,tag3 @ 93
> }
> }
> }
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: batch_mutate silently failing in Cassandra
Posted by Aaron Morton <aa...@thelastpickle.com>.
Turn up the logging to DEBUG level (in config/log4-server.properties) and check that you are sending what you think you are.
Aaron
On 08 Sep, 2010,at 02:11 AM, Lucas Nodine <lu...@gmail.com> wrote:
Hello all,
I have posted the following to Stackoverflow, but thought that I would also try the list. If you have any suggestions, please let me know
I am working with Cassandra 0.6.5 using the thrift interface. I am trying to use the batch_mutate method call, however, when I execute it, I receive no error message. This leads me to believe it worked. When I check using the CLI, there is nothing there. Is there something wrong with my code or format of the mutation_map that anyone can see? Any ideas?
Thanks in advance,
LN
public void Update(string keyspace, Common.NetworkPackage.MetaAsset ma)
{
Dictionary<string, Dictionary<string, List<Mutation>>> package;
Dictionary<string, List<Mutation>> packageEntry;
Dictionary<string, object>.Enumerator en;
List<Mutation> mutList;
Mutation mut;
DateTime now = DateTime.Now;
if(!ma.Fields.ContainsKey("$guid"))
throw new ArgumentException("The field $guid is not present");
mutList = new List<Mutation>();
en = ma.Fields.GetEnumerator();
while(en.MoveNext())
{
if (en.Current.Value == null)
continue;
mut = new Mutation();
mut.Column_or_supercolumn = new ColumnOrSuperColumn();
mut.Column_or_supercolumn.Column = new Column();
mut.Column_or_supercolumn.Column.Name = _utf8.GetBytes(en.Current.Key);
if (en.Current.Value == null)
mut.Column_or_supercolumn.Column.Value = null;
else
mut.Column_or_supercolumn.Column.Value = ToBytes(en.Current.Value);
mut.Column_or_supercolumn.Column.Timestamp = Utilities.Timestamp(now);
mutList.Add(mut);
}
packageEntry = new Dictionary<string, List<Mutation>>();
packageEntry.Add("MetaAsset", mutList);
package = new Dictionary<string, Dictionary<string, List<Mutation>>>();
package.Add((string)ma.Fields["$guid"], packageEntry);
Console.WriteLine(Utilities.ExportBulkMutate("LawOffice", package));
_client.batch_mutate(keyspace, package, ConsistencyLevel.QUORUM);
}
The above code produces (columns are name:value @ timestamp, value consists of a type:and a representation of the actual value):
LawOffice : {
Row=08469fba50f448be8943614abd059d10 : {
CF=MetaAsset : {
$guid : String:08469fba50f448be8943614abd059d10 @ 93
$creator : String:Lucas @ 93
$previousversion : String:00000000000000000000000000000000 @ 93
$nextversion : String:00000000000000000000000000000000 @ 93
$etag : String:0 @ 93
$length : Int32:123456789 @ 93
$extension : String:.odt @ 93
$created : DateTime:90 @ 93
$modified : DateTime:90 @ 93
$lastaccess : DateTime:90 @ 93
$title : String:Title @ 93
$tags : List`1:tag1,tag2,tag3 @ 93
}
}
}