You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Alicia Leong <lc...@gmail.com> on 2013/03/29 16:42:39 UTC
Cassandra/MapReduce ‘Data Locality’
Hi All,
CfSplit that highlighted in RED* **, *in *d2t0053g*
But why it being submitted to *d2t0051g *not *d2t0053g ??*
Is this normal? Does this matter? In this case is no longer ‘Data Locality’
correct ?
I’m using hadoop-1.1.2 & apache-cassandra-1.2.3.
TokenRange (1) >> 127605887595351923798765477786913079296 => 0
TokenRange (2) >> 85070591730234615865843651857942052864 =>
127605887595351923798765477786913079296
TokenRange (3) >> 42535295865117307932921825928971026432 =>
85070591730234615865843651857942052864
TokenRange (4) >> 0 => 42535295865117307932921825928971026432
ColumnFamilySplit((127605887595351923798765477786913079296, '-1] @[d2t0050g
])
ColumnFamilySplit((-1, '0] @[d2t0050g])
ColumnFamilySplit((85070591730234615865843651857942052864,
'127605887595351923798765477786913079296] @[d2t0053g])
ColumnFamilySplit((42535295865117307932921825928971026432,
'85070591730234615865843651857942052864] @[d2t0052g])
ColumnFamilySplit((0, '42535295865117307932921825928971026432] @[d2t0051g])
--------*RF1*-------
*d2t0050g *
KeyRange(start_token:127605887595351923798765477786913079296, end_token:-1,
count:4096)
*d2t0051g*
KeyRange(start_token:85070591730234615865843651857942052864,
end_token:127605887595351923798765477786913079296, count:4096)
Rowkey:3; columnvalue=Critics Choice Awards from
ColumnFamilySplit((85070591730234615865843651857942052864,
'127605887595351923798765477786913079296] @[d2t0053g])
KeyRange(start_token:117356732921465116845890410746976120467,
end_token:127605887595351923798765477786913079296, count:4096)
KeyRange(start_token:0, end_token:42535295865117307932921825928971026432,
count:4096)
Rowkey:1; columnvalue=Academy Awards from ColumnFamilySplit((0,
'42535295865117307932921825928971026432] @[d2t0051g])
Rowkey 2: columnvalue=Golden Globe Awards from ColumnFamilySplit((0,
'42535295865117307932921825928971026432] @[d2t0051g])
KeyRange(start_token:19847720572362509985402305765727304993,
end_token:42535295865117307932921825928971026432, count:4096)
*d2t0052g*
KeyRange(start_token:42535295865117307932921825928971026432,
end_token:85070591730234615865843651857942052864, count:4096)
KeyRange(start_token:-1, end_token:0, count:4096)
*d2t0053g*
Nil
Thanks in advance.
Re: Cassandra/MapReduce ‘Data Locality’
Posted by Alicia Leong <lc...@gmail.com>.
I have 4 Cassandra nodes that also installed with Datanode & TaskTracker.
This log printed at the console, when I execute hadoop jar
TokenRange (1) >> 127605887595351923798765477786913079296 => 0
TokenRange (2) >> 85070591730234615865843651857942052864 =>
127605887595351923798765477786913079296
TokenRange (3) >> 42535295865117307932921825928971026432 =>
85070591730234615865843651857942052864
TokenRange (4) >> 0 => 42535295865117307932921825928971026432
ColumnFamilySplit((127605887595351923798765477786913079296, '-1] @[d2t0050g
])
ColumnFamilySplit((-1, '0] @[d2t0050g])
ColumnFamilySplit((85070591730234615865843651857942052864,
'127605887595351923798765477786913079296] @[d2t0053g])
ColumnFamilySplit((42535295865117307932921825928971026432,
'85070591730234615865843651857942052864] @[d2t0052g])
ColumnFamilySplit((0, '42535295865117307932921825928971026432] @[d2t0051g])
Below logs I grabbed it at each Datanode map log folder.
In my mapper map method
System.out.println("Rowkey " + ByteBufferUtil.toInt(key) + ":" + name + "="
+ value + " from " + context.getInputSplit());
--------*RF1*-------
*d2t0050g*
KeyRange(start_token:127605887595351923798765477786913079296, end_token:-1,
count:4096)
*d2t0051g*
KeyRange(start_token:85070591730234615865843651857942052864,
end_token:127605887595351923798765477786913079296, count:4096)
Rowkey:3; columnvalue=Critics Choice Awards from
ColumnFamilySplit((85070591730234615865843651857942052864,
'127605887595351923798765477786913079296] @[d2t0053g])
KeyRange(start_token:117356732921465116845890410746976120467,
end_token:127605887595351923798765477786913079296, count:4096)
KeyRange(start_token:0, end_token:42535295865117307932921825928971026432,
count:4096)
Rowkey:1; columnvalue=Academy Awards from ColumnFamilySplit((0,
'42535295865117307932921825928971026432] @[d2t0051g])
Rowkey 2: columnvalue=Golden Globe Awards from ColumnFamilySplit((0,
'42535295865117307932921825928971026432] @[d2t0051g])
KeyRange(start_token:19847720572362509985402305765727304993,
end_token:42535295865117307932921825928971026432, count:4096)
*d2t0052g*
KeyRange(start_token:42535295865117307932921825928971026432,
end_token:85070591730234615865843651857942052864, count:4096)
KeyRange(start_token:-1, end_token:0, count:4096)
*d2t0053g*
Nil
On Sun, Mar 31, 2013 at 6:26 PM, aaron morton <aa...@thelastpickle.com>wrote:
> > ColumnFamilySplit((85070591730234615865843651857942052864,
> '127605887595351923798765477786913079296] @[d2t0053g])
> Can you provide some more information on where these log lines are from
> and what you did to get them ?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 29/03/2013, at 9:12 PM, Alicia Leong <lc...@gmail.com> wrote:
>
> > Hi All,
> >
> > CfSplit that highlighted in RED , in d2t0053g
> >
> > But why it being submitted to d2t0051g not d2t0053g ??
> >
> > Is this normal? Does this matter? In this case is no longer ‘Data
> Locality’ correct ?
> >
> >
> >
> > I’m using hadoop-1.1.2 & apache-cassandra-1.2.3.
> >
> > TokenRange (1) >> 127605887595351923798765477786913079296 => 0
> >
> > TokenRange (2) >> 85070591730234615865843651857942052864 =>
> 127605887595351923798765477786913079296
> >
> > TokenRange (3) >> 42535295865117307932921825928971026432 =>
> 85070591730234615865843651857942052864
> >
> > TokenRange (4) >> 0 => 42535295865117307932921825928971026432
> >
> > ColumnFamilySplit((127605887595351923798765477786913079296, '-1]
> @[d2t0050g])
> >
> > ColumnFamilySplit((-1, '0] @[d2t0050g])
> >
> > ColumnFamilySplit((85070591730234615865843651857942052864,
> '127605887595351923798765477786913079296] @[d2t0053g])
> >
> > ColumnFamilySplit((42535295865117307932921825928971026432,
> '85070591730234615865843651857942052864] @[d2t0052g])
> >
> > ColumnFamilySplit((0, '42535295865117307932921825928971026432]
> @[d2t0051g])
> >
> >
> > --------RF1-------
> >
> > d2t0050g
> >
> > KeyRange(start_token:127605887595351923798765477786913079296,
> end_token:-1, count:4096)
> >
> > d2t0051g
> >
> > KeyRange(start_token:85070591730234615865843651857942052864,
> end_token:127605887595351923798765477786913079296, count:4096)
> >
> > Rowkey:3; columnvalue=Critics Choice Awards from
> ColumnFamilySplit((85070591730234615865843651857942052864,
> '127605887595351923798765477786913079296] @[d2t0053g])
> >
> > KeyRange(start_token:117356732921465116845890410746976120467,
> end_token:127605887595351923798765477786913079296, count:4096)
> >
> > KeyRange(start_token:0,
> end_token:42535295865117307932921825928971026432, count:4096)
> >
> > Rowkey:1; columnvalue=Academy Awards from ColumnFamilySplit((0,
> '42535295865117307932921825928971026432] @[d2t0051g])
> >
> > Rowkey 2: columnvalue=Golden Globe Awards from ColumnFamilySplit((0,
> '42535295865117307932921825928971026432] @[d2t0051g])
> >
> > KeyRange(start_token:19847720572362509985402305765727304993,
> end_token:42535295865117307932921825928971026432, count:4096)
> >
> > d2t0052g
> >
> > KeyRange(start_token:42535295865117307932921825928971026432,
> end_token:85070591730234615865843651857942052864, count:4096)
> >
> > KeyRange(start_token:-1, end_token:0, count:4096)
> >
> > d2t0053g
> >
> > Nil
> >
> >
> >
> >
> >
> >
> >
> > Thanks in advance.
> >
>
>
Re: Cassandra/MapReduce ‘Data Locality’
Posted by aaron morton <aa...@thelastpickle.com>.
> ColumnFamilySplit((85070591730234615865843651857942052864, '127605887595351923798765477786913079296] @[d2t0053g])
Can you provide some more information on where these log lines are from and what you did to get them ?
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 29/03/2013, at 9:12 PM, Alicia Leong <lc...@gmail.com> wrote:
> Hi All,
>
> CfSplit that highlighted in RED , in d2t0053g
>
> But why it being submitted to d2t0051g not d2t0053g ??
>
> Is this normal? Does this matter? In this case is no longer ‘Data Locality’ correct ?
>
>
>
> I’m using hadoop-1.1.2 & apache-cassandra-1.2.3.
>
> TokenRange (1) >> 127605887595351923798765477786913079296 => 0
>
> TokenRange (2) >> 85070591730234615865843651857942052864 => 127605887595351923798765477786913079296
>
> TokenRange (3) >> 42535295865117307932921825928971026432 => 85070591730234615865843651857942052864
>
> TokenRange (4) >> 0 => 42535295865117307932921825928971026432
>
> ColumnFamilySplit((127605887595351923798765477786913079296, '-1] @[d2t0050g])
>
> ColumnFamilySplit((-1, '0] @[d2t0050g])
>
> ColumnFamilySplit((85070591730234615865843651857942052864, '127605887595351923798765477786913079296] @[d2t0053g])
>
> ColumnFamilySplit((42535295865117307932921825928971026432, '85070591730234615865843651857942052864] @[d2t0052g])
>
> ColumnFamilySplit((0, '42535295865117307932921825928971026432] @[d2t0051g])
>
>
> --------RF1-------
>
> d2t0050g
>
> KeyRange(start_token:127605887595351923798765477786913079296, end_token:-1, count:4096)
>
> d2t0051g
>
> KeyRange(start_token:85070591730234615865843651857942052864, end_token:127605887595351923798765477786913079296, count:4096)
>
> Rowkey:3; columnvalue=Critics Choice Awards from ColumnFamilySplit((85070591730234615865843651857942052864, '127605887595351923798765477786913079296] @[d2t0053g])
>
> KeyRange(start_token:117356732921465116845890410746976120467, end_token:127605887595351923798765477786913079296, count:4096)
>
> KeyRange(start_token:0, end_token:42535295865117307932921825928971026432, count:4096)
>
> Rowkey:1; columnvalue=Academy Awards from ColumnFamilySplit((0, '42535295865117307932921825928971026432] @[d2t0051g])
>
> Rowkey 2: columnvalue=Golden Globe Awards from ColumnFamilySplit((0, '42535295865117307932921825928971026432] @[d2t0051g])
>
> KeyRange(start_token:19847720572362509985402305765727304993, end_token:42535295865117307932921825928971026432, count:4096)
>
> d2t0052g
>
> KeyRange(start_token:42535295865117307932921825928971026432, end_token:85070591730234615865843651857942052864, count:4096)
>
> KeyRange(start_token:-1, end_token:0, count:4096)
>
> d2t0053g
>
> Nil
>
>
>
>
>
>
>
> Thanks in advance.
>