You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Dan Di Spaltro <da...@gmail.com> on 2009/09/30 20:47:30 UTC
Binary Loading Question
I was trying to binary load some data into an instance and ran into a
null exception in the process. So I began looking at the code, and it
looks like around Table.java:653 of trunk, that piece of code doesn't
really make sense. I don't understand the internals of Cassandra all
that well, but this doesn't seem right. The null exception came from
null cfStore.
Either way I am just trying to use the binarymemtable option to load
values into cassandra, since I have about 240 million rows to load.
{
ColumnFamilyStore cfStore =
columnFamilyStores_.get(new String(column.name(), "UTF-8"));
cfStore.applyBinary(key, column.value());
}
--
Dan Di Spaltro
Re: Binary Loading Question
Posted by Dan Di Spaltro <da...@gmail.com>.
Chris,
Sorry for the delay. Jbellis figured it out, I am using the OPP and
during serialization the Token was null and throwing.
I will say after looking through the contrib code, the BMT is pretty
odd and especially the way it gets serialized with indexes; then the
payload dropped into the value and the column name is loaded with the
actual ColumnFamily name; however, we have about half a billion points
to migrate and ended up getting like 35k sec from one pretty weak
machine.
Thanks for the help,
On Wed, Sep 30, 2009 at 7:15 PM, Chris Goffinet <cg...@chrisgoffinet.com> wrote:
> It would be helpful if you could attach your example. Also did you look at
> the example in contrib by chance?
>
>
> On Sep 30, 2009, at 2:05 PM, Dan Di Spaltro wrote:
>
>> Okay i get what its doing now... its pretty confusing until I saw
>> jbellis' comments. now when i flush binary it is throwing on the
>> key.toString line because the token is null. When i setup the
>> rowmutation on the client i used StringToken as opposed to big integer
>> but thats because i am not sure what to put there so i used the tokens
>> chose by cassandra
>>
>> On 9/30/09, Chris Goffinet <cg...@chrisgoffinet.com> wrote:
>>>
>>> It's very jank. I kinda understand where they were coming from on that.
>>> Let
>>> me think through it some more and open a ticket to improve some of that
>>> code.
>>>
>>>
>>> On Wed, Sep 30, 2009 at 12:22 PM, Jonathan Ellis <jb...@gmail.com>
>>> wrote:
>>>
>>>> So it looks like it takes multiple rows, one per "column" in the
>>>> rowmutation.
>>>>
>>>> Which is confusing and weird if you ask me.
>>>>
>>>> On Wed, Sep 30, 2009 at 1:57 PM, Jonathan Ellis <jb...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Yeah, just eyeballing it
>>>>>
>>>>> ColumnFamilyStore cfStore =
>>>>> columnFamilyStores_.get(columnFamily.name());
>>>>>
>>>>> would make more sense to me.
>>>>>
>>>>> Got any insight for us, Chris? :)
>>>>>
>>>>> -Jonathan
>>>>>
>>>>> On Wed, Sep 30, 2009 at 1:47 PM, Dan Di Spaltro
>>>>> <da...@gmail.com>
>>>>
>>>> wrote:
>>>>>>
>>>>>> I was trying to binary load some data into an instance and ran into a
>>>>>> null exception in the process. So I began looking at the code, and it
>>>>>> looks like around Table.java:653 of trunk, that piece of code doesn't
>>>>>> really make sense. I don't understand the internals of Cassandra all
>>>>>> that well, but this doesn't seem right. The null exception came from
>>>>>> null cfStore.
>>>>>>
>>>>>> Either way I am just trying to use the binarymemtable option to load
>>>>>> values into cassandra, since I have about 240 million rows to load.
>>>>>>
>>>>>> {
>>>>>> ColumnFamilyStore cfStore =
>>>>>> columnFamilyStores_.get(new String(column.name(), "UTF-8"));
>>>>>> cfStore.applyBinary(key, column.value());
>>>>>> }
>>>>>>
>>>>>> --
>>>>>> Dan Di Spaltro
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Dan Di Spaltro
>
>
--
Dan Di Spaltro
Re: Binary Loading Question
Posted by Chris Goffinet <cg...@chrisgoffinet.com>.
It would be helpful if you could attach your example. Also did you
look at the example in contrib by chance?
On Sep 30, 2009, at 2:05 PM, Dan Di Spaltro wrote:
> Okay i get what its doing now... its pretty confusing until I saw
> jbellis' comments. now when i flush binary it is throwing on the
> key.toString line because the token is null. When i setup the
> rowmutation on the client i used StringToken as opposed to big integer
> but thats because i am not sure what to put there so i used the tokens
> chose by cassandra
>
> On 9/30/09, Chris Goffinet <cg...@chrisgoffinet.com> wrote:
>> It's very jank. I kinda understand where they were coming from on
>> that. Let
>> me think through it some more and open a ticket to improve some of
>> that
>> code.
>>
>>
>> On Wed, Sep 30, 2009 at 12:22 PM, Jonathan Ellis
>> <jb...@gmail.com> wrote:
>>
>>> So it looks like it takes multiple rows, one per "column" in the
>>> rowmutation.
>>>
>>> Which is confusing and weird if you ask me.
>>>
>>> On Wed, Sep 30, 2009 at 1:57 PM, Jonathan Ellis
>>> <jb...@gmail.com> wrote:
>>>> Yeah, just eyeballing it
>>>>
>>>> ColumnFamilyStore cfStore =
>>>> columnFamilyStores_.get(columnFamily.name());
>>>>
>>>> would make more sense to me.
>>>>
>>>> Got any insight for us, Chris? :)
>>>>
>>>> -Jonathan
>>>>
>>>> On Wed, Sep 30, 2009 at 1:47 PM, Dan Di Spaltro
>>>> <da...@gmail.com>
>>> wrote:
>>>>> I was trying to binary load some data into an instance and ran
>>>>> into a
>>>>> null exception in the process. So I began looking at the code,
>>>>> and it
>>>>> looks like around Table.java:653 of trunk, that piece of code
>>>>> doesn't
>>>>> really make sense. I don't understand the internals of
>>>>> Cassandra all
>>>>> that well, but this doesn't seem right. The null exception came
>>>>> from
>>>>> null cfStore.
>>>>>
>>>>> Either way I am just trying to use the binarymemtable option to
>>>>> load
>>>>> values into cassandra, since I have about 240 million rows to
>>>>> load.
>>>>>
>>>>> {
>>>>> ColumnFamilyStore cfStore =
>>>>> columnFamilyStores_.get(new String(column.name(), "UTF-8"));
>>>>> cfStore.applyBinary(key, column.value());
>>>>> }
>>>>>
>>>>> --
>>>>> Dan Di Spaltro
>>>>>
>>>>
>>>
>>
>
>
> --
> Dan Di Spaltro
Re: Binary Loading Question
Posted by Dan Di Spaltro <da...@gmail.com>.
Okay i get what its doing now... its pretty confusing until I saw
jbellis' comments. now when i flush binary it is throwing on the
key.toString line because the token is null. When i setup the
rowmutation on the client i used StringToken as opposed to big integer
but thats because i am not sure what to put there so i used the tokens
chose by cassandra
On 9/30/09, Chris Goffinet <cg...@chrisgoffinet.com> wrote:
> It's very jank. I kinda understand where they were coming from on that. Let
> me think through it some more and open a ticket to improve some of that
> code.
>
>
> On Wed, Sep 30, 2009 at 12:22 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>
>> So it looks like it takes multiple rows, one per "column" in the
>> rowmutation.
>>
>> Which is confusing and weird if you ask me.
>>
>> On Wed, Sep 30, 2009 at 1:57 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>> > Yeah, just eyeballing it
>> >
>> > ColumnFamilyStore cfStore =
>> > columnFamilyStores_.get(columnFamily.name());
>> >
>> > would make more sense to me.
>> >
>> > Got any insight for us, Chris? :)
>> >
>> > -Jonathan
>> >
>> > On Wed, Sep 30, 2009 at 1:47 PM, Dan Di Spaltro
>> > <da...@gmail.com>
>> wrote:
>> >> I was trying to binary load some data into an instance and ran into a
>> >> null exception in the process. So I began looking at the code, and it
>> >> looks like around Table.java:653 of trunk, that piece of code doesn't
>> >> really make sense. I don't understand the internals of Cassandra all
>> >> that well, but this doesn't seem right. The null exception came from
>> >> null cfStore.
>> >>
>> >> Either way I am just trying to use the binarymemtable option to load
>> >> values into cassandra, since I have about 240 million rows to load.
>> >>
>> >> {
>> >> ColumnFamilyStore cfStore =
>> >> columnFamilyStores_.get(new String(column.name(), "UTF-8"));
>> >> cfStore.applyBinary(key, column.value());
>> >> }
>> >>
>> >> --
>> >> Dan Di Spaltro
>> >>
>> >
>>
>
--
Dan Di Spaltro
Re: Binary Loading Question
Posted by Chris Goffinet <cg...@chrisgoffinet.com>.
It's very jank. I kinda understand where they were coming from on that. Let
me think through it some more and open a ticket to improve some of that
code.
On Wed, Sep 30, 2009 at 12:22 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> So it looks like it takes multiple rows, one per "column" in the
> rowmutation.
>
> Which is confusing and weird if you ask me.
>
> On Wed, Sep 30, 2009 at 1:57 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> > Yeah, just eyeballing it
> >
> > ColumnFamilyStore cfStore =
> > columnFamilyStores_.get(columnFamily.name());
> >
> > would make more sense to me.
> >
> > Got any insight for us, Chris? :)
> >
> > -Jonathan
> >
> > On Wed, Sep 30, 2009 at 1:47 PM, Dan Di Spaltro <da...@gmail.com>
> wrote:
> >> I was trying to binary load some data into an instance and ran into a
> >> null exception in the process. So I began looking at the code, and it
> >> looks like around Table.java:653 of trunk, that piece of code doesn't
> >> really make sense. I don't understand the internals of Cassandra all
> >> that well, but this doesn't seem right. The null exception came from
> >> null cfStore.
> >>
> >> Either way I am just trying to use the binarymemtable option to load
> >> values into cassandra, since I have about 240 million rows to load.
> >>
> >> {
> >> ColumnFamilyStore cfStore =
> >> columnFamilyStores_.get(new String(column.name(), "UTF-8"));
> >> cfStore.applyBinary(key, column.value());
> >> }
> >>
> >> --
> >> Dan Di Spaltro
> >>
> >
>
Re: Binary Loading Question
Posted by Jonathan Ellis <jb...@gmail.com>.
So it looks like it takes multiple rows, one per "column" in the rowmutation.
Which is confusing and weird if you ask me.
On Wed, Sep 30, 2009 at 1:57 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> Yeah, just eyeballing it
>
> ColumnFamilyStore cfStore =
> columnFamilyStores_.get(columnFamily.name());
>
> would make more sense to me.
>
> Got any insight for us, Chris? :)
>
> -Jonathan
>
> On Wed, Sep 30, 2009 at 1:47 PM, Dan Di Spaltro <da...@gmail.com> wrote:
>> I was trying to binary load some data into an instance and ran into a
>> null exception in the process. So I began looking at the code, and it
>> looks like around Table.java:653 of trunk, that piece of code doesn't
>> really make sense. I don't understand the internals of Cassandra all
>> that well, but this doesn't seem right. The null exception came from
>> null cfStore.
>>
>> Either way I am just trying to use the binarymemtable option to load
>> values into cassandra, since I have about 240 million rows to load.
>>
>> {
>> ColumnFamilyStore cfStore =
>> columnFamilyStores_.get(new String(column.name(), "UTF-8"));
>> cfStore.applyBinary(key, column.value());
>> }
>>
>> --
>> Dan Di Spaltro
>>
>
Re: Binary Loading Question
Posted by Jonathan Ellis <jb...@gmail.com>.
Yeah, just eyeballing it
ColumnFamilyStore cfStore =
columnFamilyStores_.get(columnFamily.name());
would make more sense to me.
Got any insight for us, Chris? :)
-Jonathan
On Wed, Sep 30, 2009 at 1:47 PM, Dan Di Spaltro <da...@gmail.com> wrote:
> I was trying to binary load some data into an instance and ran into a
> null exception in the process. So I began looking at the code, and it
> looks like around Table.java:653 of trunk, that piece of code doesn't
> really make sense. I don't understand the internals of Cassandra all
> that well, but this doesn't seem right. The null exception came from
> null cfStore.
>
> Either way I am just trying to use the binarymemtable option to load
> values into cassandra, since I have about 240 million rows to load.
>
> {
> ColumnFamilyStore cfStore =
> columnFamilyStores_.get(new String(column.name(), "UTF-8"));
> cfStore.applyBinary(key, column.value());
> }
>
> --
> Dan Di Spaltro
>