You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Mck SembWever (JIRA)" <ji...@apache.org> on 2011/09/07 19:45:09 UTC
[jira] [Created] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever
ColumnFormatRecordReader loops forever
--------------------------------------
Key: CASSANDRA-3150
URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
Project: Cassandra
Issue Type: Bug
Components: Hadoop
Affects Versions: 0.8.4
Reporter: Mck SembWever
Assignee: Mck SembWever
Priority: Critical
>From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
{quote}
bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
bq. CFIF's inputSplitSize=196608
bq. 3 map tasks (from 4013) is still running after read 25 million rows.
bq. Can this be a bug in StorageService.getSplits(..) ?
getSplits looks pretty foolproof to me but I guess we'd need to add
more debug logging to rule out a bug there for sure.
I guess the main alternative would be a bug in the recordreader paging.
{quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099243#comment-13099243 ]
Jonathan Ellis commented on CASSANDRA-3150:
-------------------------------------------
the next startToken always comes from the most recently returned range (s.t. start < range <= end), so unless there's a bad bug in get_range_slices it can't ever sort after endToken.
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mck SembWever updated CASSANDRA-3150:
-------------------------------------
Summary: ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack) (was: ColumnFormatRecordReader loops forever)
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mck SembWever resolved CASSANDRA-3150.
--------------------------------------
Resolution: Cannot Reproduce
Fix Version/s: 0.8.6
An upgrade to 0.8.6 /seems/ to have fixed this.
I saw one map task (CFRR) reach 150% but nothing like i was seeing before.
Maybe there is something else happening in the upgrade process that fixed it?
Otherwise i'm happy to close this issue for now and reopen if the problem exacerbates again.
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099234#comment-13099234 ]
Mck SembWever edited comment on CASSANDRA-3150 at 9/7/11 7:17 PM:
------------------------------------------------------------------
Here keyRange is startToken to split.getEndToken()
startToken is updated each iterate to the last row read (each iterate is batchRowCount rows).
What happens if split.getEndToken() doesn't correspond to any of the rowKeys?
To me it reads that startToken will hop over split.getEndToken() and get_range_slices(..) will start returning wrapping ranges. This will still return rows and so the iteration will continue, now forever.
The only way out for this code today is a) startToken equals split.getEndToken(), or b) get_range_slices(..) is called with startToken equals split.getEndToken() OR a gap so small there exists no rows in between.
was (Author: michaelsembwever):
Here keyRange is startToken to split.getEndToken()
startToken is updated each iterate to the last row read (each iterate is batchRowCount rows).
What happens is split.getEndToken() doesn't correspond to any of the rowKeys?
To me it reads that startToken will hop over split.getEndToken() and get_rage_slices(..) will start returning wrapping ranges. This will still return rows and so the iteration will continue, now forever.
The only way out for this code today is a) startToken equals split.getEndToken(), or b) get_range_slices(..) is called with startToken equals split.getEndToken() OR a gap so small there exists no rows in between.
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099254#comment-13099254 ]
Mck SembWever commented on CASSANDRA-3150:
------------------------------------------
What about the case where tokens of different length exist.
I don't know if this is actually possible but from
{noformat}
Address Status State Load Owns Token
Token(bytes[76118303760208547436305468318170713656])
152.90.241.22 Up Normal 270.46 GB 33.33% Token(bytes[30303030303031333131313739353337303038d4e7f72db2ed11e09d7c68b59973a5d8])
152.90.241.24 Up Normal 247.89 GB 33.33% Token(bytes[303030303030313331323631393735313231381778518cc00711e0acb968b59973a5d8])
152.90.241.23 Up Normal 1.1 TB 33.33% Token(bytes[76118303760208547436305468318170713656])
{noformat}
you see the real tokens are very long compared to the initial_tokens the cluster was configured with. (The two long tokens has since been moved, and to note the load on .23 never decreased to ~300GB as it should have...).
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099943#comment-13099943 ]
Mck SembWever commented on CASSANDRA-3150:
------------------------------------------
I'll try and put debug in so i can get a log of get_slice_range calls from CFRR... (this may take some days)
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114702#comment-13114702 ]
T Jake Luciani commented on CASSANDRA-3150:
-------------------------------------------
bq. The problematic split ends up being ...
So you are saying that split keeps getting called over and over? I dont see that in the log or does it hang?
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, fullscan-example1.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114196#comment-13114196 ]
Mck SembWever commented on CASSANDRA-3150:
------------------------------------------
Yes i see now an CFRR that went to 600%. But it's still a long way from problematic.
My understanding of how the sampled keys is generated is it all happens when a sstable is read.
If a restart or a (manual) compact doesn't help, why did an upgrade help?
What can i investigate to provide more debug here?
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114220#comment-13114220 ]
Mck SembWever edited comment on CASSANDRA-3150 at 9/25/11 11:27 AM:
--------------------------------------------------------------------
Here's debug from a "full scan" job. It scans data over a full year (and since the cluster's ring range only holds 3 months of data such a job guarantees a full scan).
In the debug you see the splits.
{{`nodetool ring`}} gives
{noformat}Address DC Rack Status State Load Owns Token
Token(bytes[55555555555555555555555555555554])
152.90.241.22 DC1 RAC1 Up Normal 16.65 GB 33.33% Token(bytes[00])
152.90.241.23 DC2 RAC1 Up Normal 63.22 GB 33.33% Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa])
152.90.241.24 DC1 RAC1 Up Normal 72.4 KB 33.33% Token(bytes[55555555555555555555555555555554])
{noformat}
The problematic split ends up being
{noformat}ColumnFamilySplit{startToken='0528cbe0b2b5ff6b816c68b59973bcbc', endToken='2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', dataNodes=[cassandra02.finn.no]}{noformat} This is the split that's receiving new data (5-10k rows/second).
was (Author: michaelsembwever):
Here's debug from a "full scan" job. It scans data over a full year (and since the cluster's ring range only holds 3 months of data such a job guarantees a full scan).
In the debug you see the splits.
{{`nodetool ring`}} gives
{noformat}Address DC Rack Status State Load Owns Token
Token(bytes[55555555555555555555555555555554])
152.90.241.22 DC1 RAC1 Up Normal 16.65 GB 33.33% Token(bytes[00])
152.90.241.23 DC2 RAC1 Up Normal 63.22 GB 33.33% Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa])
152.90.241.24 DC1 RAC1 Up Normal 72.4 KB 33.33% Token(bytes[55555555555555555555555555555554])
{noformat}
The problematic split ends up being
{noformat}ColumnFamilySplit{startToken='0528cbe0b2b5ff6b816c68b59973bcbc', endToken='2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', dataNodes=[cassandra02.finn.no]}{noformat}
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, fullscan-example1.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mck SembWever updated CASSANDRA-3150:
-------------------------------------
Attachment: Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099217#comment-13099217 ]
Jonathan Ellis commented on CASSANDRA-3150:
-------------------------------------------
The start==end check on 234 is a special case, because start==end is a wrapping Range.
The main "stop when we're done logic" is this:
{code}
rows = client.get_range_slices(new ColumnParent(cfName),
predicate,
keyRange,
consistencyLevel);
// nothing new? reached the end
if (rows.isEmpty())
{
rows = null;
return;
}
{code}
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mck SembWever updated CASSANDRA-3150:
-------------------------------------
Attachment: Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
whack)
Posted by "Mck SembWever (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139728#comment-13139728 ]
Mck SembWever edited comment on CASSANDRA-3150 at 10/30/11 7:46 PM:
--------------------------------------------------------------------
Back after an upgrade to cassandra-1.0.0
Example job start logs{noformat}[ INFO] 20:39:21 Restricting input range: 3589a548d20f80a7b41368b59973bcbc -- 36f0bedaf02a49a3b41368b59973bcbc [] at no.finntech.countstats.reduce.rolled.internal.GenericCountAggregation.configureCountToAggregateMapper(GenericCountAggregation.java:222)
[ INFO] 20:39:21 Corresponding time range is Sun Oct 30 00:00:00 CEST 2011 (1319925600000) -- Sun Oct 30 23:00:00 CET 2011 (1320012000000) [] at no.finntech.countstats.reduce.rolled.internal.GenericCountAggregation.configureCountToAggregateMapper(GenericCountAggregation.java:225)
[ INFO] 20:39:21 Starting AdIdCountAggregation-phase1-DAY_2011303 ( to=1320002999000)) [] at no.finntech.countstats.reduce.rolled.internal.GenericCountAggregation.run(GenericCountAggregation.java:142)
[DEBUG] 20:39:21 adding ColumnFamilySplit{startToken='3589a548d20f80a7b41368b59973bcbc', endToken='36f0bedaf02a49a3b41368b59973bcbc', dataNodes=[0.0.0.0]} [] at org.apache.cassandra.hadoop.ColumnFamilyInputFormat$SplitCallable.call(ColumnFamilyInputFormat.java:210)
{noformat}
In this split there is in fact ~40 million rows, and with a split size of 393216, it is expected ~100 splits to be returned.
I'm also very confused by the dataNodes=[0.0.0.0]. this looks to be wrong, i know that data lies on the third node: cassandra03, where-else the job is running on the first node: cassandra01.
was (Author: michaelsembwever):
Back after an upgrade to cassandra-1.0.0
Example job start logs{noformat}[ INFO] 20:39:21 Restricting input range: 3589a548d20f80a7b41368b59973bcbc -- 36f0bedaf02a49a3b41368b59973bcbc [] at no.finntech.countstats.reduce.rolled.internal.GenericCountAggregation.configureCountToAggregateMapper(GenericCountAggregation.java:222)
[ INFO] 20:39:21 Corresponding time range is Sun Oct 30 00:00:00 CEST 2011 (1319925600000) -- Sun Oct 30 23:00:00 CET 2011 (1320012000000) [] at no.finntech.countstats.reduce.rolled.internal.GenericCountAggregation.configureCountToAggregateMapper(GenericCountAggregation.java:225)
[ INFO] 20:39:21 Starting AdIdCountAggregation-phase1-DAY_2011303 ( to=1320002999000)) [] at no.finntech.countstats.reduce.rolled.internal.GenericCountAggregation.run(GenericCountAggregation.java:142)
[DEBUG] 20:39:21 adding ColumnFamilySplit{startToken='3589a548d20f80a7b41368b59973bcbc', endToken='36f0bedaf02a49a3b41368b59973bcbc', dataNodes=[0.0.0.0]} [] at org.apache.cassandra.hadoop.ColumnFamilyInputFormat$SplitCallable.call(ColumnFamilyInputFormat.java:210)
{noformat}
In this split there is in fact ~40 million rows, and with a split size of 393216, it is expected ~100 splits to be returned.
I'm also very confused by the {{dataNodes=[0.0.0.0]}}. this looks to be wrong, i know that data lies on the third node: cassandra03, where-else the job is running on the first node: cassandra01.
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, fullscan-example1.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099234#comment-13099234 ]
Mck SembWever edited comment on CASSANDRA-3150 at 9/7/11 7:24 PM:
------------------------------------------------------------------
Here keyRange is startToken to split.getEndToken()
startToken is updated each iterate to the last row read (each iterate is batchRowCount rows).
What happens if split.getEndToken() doesn't correspond to any of the rowKeys?
To me it reads that startToken will hop over split.getEndToken() and get_range_slices(..) will start querying against wrapping ranges. This will still return rows and so the iteration will continue, now forever.
The only way out for this code today is a) startToken equals split.getEndToken(), or b) get_range_slices(..) is called with startToken equals split.getEndToken() OR a gap so small there exists no rows in between.
was (Author: michaelsembwever):
Here keyRange is startToken to split.getEndToken()
startToken is updated each iterate to the last row read (each iterate is batchRowCount rows).
What happens if split.getEndToken() doesn't correspond to any of the rowKeys?
To me it reads that startToken will hop over split.getEndToken() and get_range_slices(..) will start returning wrapping ranges. This will still return rows and so the iteration will continue, now forever.
The only way out for this code today is a) startToken equals split.getEndToken(), or b) get_range_slices(..) is called with startToken equals split.getEndToken() OR a gap so small there exists no rows in between.
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
whack)
Posted by "Mck SembWever (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115278#comment-13115278 ]
Mck SembWever edited comment on CASSANDRA-3150 at 9/27/11 7:14 AM:
-------------------------------------------------------------------
No. (the title is incorrect).
That split contains millions of rows, despite StorageService.getSplits(..) being called with splitSize 393216.
In the worse situations the split has 20 millions rows in it. These were all valid and had to be all processed for the hadoop job to produce accurate results.
was (Author: michaelsembwever):
No. (the title is incorrect).
That split contains millions of rows, despite StorageService.getSplits(..) being called with splitSize 393216.
In the worse situations the split has 20 millions rows in it. This were all valid are had to be process for the hadoop job to produce accurate results.
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, fullscan-example1.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113326#comment-13113326 ]
Mck SembWever edited comment on CASSANDRA-3150 at 9/23/11 11:04 AM:
--------------------------------------------------------------------
This is persisting to be a problem. And continuously gets worse through the day.
I can see like half my cf being read from one split.
It doesn't matter if it's ByteOrderingPartition or RandomPartitioner.
I've attached two screenshots from hadoop webpages (you see 36million map input records when cassandra.input.split.size is set to 393216).
was (Author: michaelsembwever):
This is persisting to be a problem. And continuously gets worse through the day.
I can see like half my cf being read from one split.
It doesn't matter if it's ByteOrderingPartition or RandomPartitioner.
I've attached two screenshots from hadoop webpages.
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102062#comment-13102062 ]
Mck SembWever edited comment on CASSANDRA-3150 at 9/10/11 6:04 PM:
-------------------------------------------------------------------
Debug from a task that was still running at 1200%
The initial split for this CFRR is
30303030303031333131313739353337303038d4e7f72db2ed11e09d7c68b59973a5d8 : 303030303030313331323631393735313231381778518cc00711e0acb968b59973a5d8
This job was run with
cassandra.input.split.size=196608
cassandra.range.batch.size=16000
therefore there shouldn't be more than 13 calls to get_range_slices(..) in this task. There was already 166 calls in this log.
What i can see here is that the original split for this task is just way too big and this comes from {{describe_splits(..)}}
which in turn depends on "index_interval". Reading {{StorageService.getSplits(..)}} i would guess that the split can in fact contain many more keys with the default sampling of 128. Question is how low can/should i bring index_interval (this cf can have up to 8 billion rows)?
was (Author: michaelsembwever):
Debug from a task that was still running at 1200%
The initial split for this CFRR is
30303030303031333131313739353337303038d4e7f72db2ed11e09d7c68b59973a5d8 : 303030303030313331323631393735313231381778518cc00711e0acb968b59973a5d8
This job was run with
cassandra.input.split.size=196608
cassandra.range.batch.size=16000
therefore there shouldn't be more than 13 calls to get_range_slices(..) in this task. There was already 166 calls in this log.
What i can see here is that the original split for this task is just way too big and this comes from {{describe_splits(..)}}
which in turn depends on "index_interval". Reading {{StorageService.getSplits(..)}} i would guess that the split can in fact contain many more keys with the default sampling of 128. Question is how low can/should i bring index_interval ?
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099263#comment-13099263 ]
Jonathan Ellis edited comment on CASSANDRA-3150 at 9/7/11 7:57 PM:
-------------------------------------------------------------------
If Token's Comparable implementation is broken, anything is possible. But I don't think it is. In your case, for instance, BOP is sorting the shorter token correctly since 3 < 7.
Load won't decrease until you run cleanup.
was (Author: jbellis):
If Token's Comparable implementation is broken, anything is possible. But I don't think it is. In your case, for instance, OPP is sorting the shorter token correctly since 3 < 7.
Load won't decrease until you run cleanup.
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113326#comment-13113326 ]
Mck SembWever commented on CASSANDRA-3150:
------------------------------------------
This is persisting to be a problem. And continuously gets worse through the day.
I can see like half my cf being read from one split.
It doesn't matter if it's ByteOrderingPartition or RandomPartitioner.
I've attached two screenshots from hadoop webpages.
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099281#comment-13099281 ]
Jonathan Ellis commented on CASSANDRA-3150:
-------------------------------------------
bq. so that doesn't explain this bug?
I'm afraid not.
bq. Never worked.
Then I guess your tokens still aren't balanced (and nodetool is smoking crack). It's virtually impossible to pick balanced tokens until we do CASSANDRA-2917.
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113326#comment-13113326 ]
Mck SembWever edited comment on CASSANDRA-3150 at 9/23/11 12:05 PM:
--------------------------------------------------------------------
This is persisting to be a problem. And continuously gets worse through the day.
I can see like half my cf being read from one split.
It doesn't matter if it's ByteOrderingPartition or RandomPartitioner.
I've attached two screenshots from hadoop webpages (you see 36million map input records when cassandra.input.split.size is set to 393216).
bq. ... double-check how many rows there really are in the given split range.
I can confirm that if i let these jobs run through to their completion (despite how terribly long they may take) that the results are correct. It would seem that the split ranges are incorrect (not the rows within in them).
was (Author: michaelsembwever):
This is persisting to be a problem. And continuously gets worse through the day.
I can see like half my cf being read from one split.
It doesn't matter if it's ByteOrderingPartition or RandomPartitioner.
I've attached two screenshots from hadoop webpages (you see 36million map input records when cassandra.input.split.size is set to 393216).
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "Mck SembWever (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139728#comment-13139728 ]
Mck SembWever commented on CASSANDRA-3150:
------------------------------------------
Back after an upgrade to cassandra-1.0.0
Example job start logs{noformat}[ INFO] 20:39:21 Restricting input range: 3589a548d20f80a7b41368b59973bcbc -- 36f0bedaf02a49a3b41368b59973bcbc [] at no.finntech.countstats.reduce.rolled.internal.GenericCountAggregation.configureCountToAggregateMapper(GenericCountAggregation.java:222)
[ INFO] 20:39:21 Corresponding time range is Sun Oct 30 00:00:00 CEST 2011 (1319925600000) -- Sun Oct 30 23:00:00 CET 2011 (1320012000000) [] at no.finntech.countstats.reduce.rolled.internal.GenericCountAggregation.configureCountToAggregateMapper(GenericCountAggregation.java:225)
[ INFO] 20:39:21 Starting AdIdCountAggregation-phase1-DAY_2011303 ( to=1320002999000)) [] at no.finntech.countstats.reduce.rolled.internal.GenericCountAggregation.run(GenericCountAggregation.java:142)
[DEBUG] 20:39:21 adding ColumnFamilySplit{startToken='3589a548d20f80a7b41368b59973bcbc', endToken='36f0bedaf02a49a3b41368b59973bcbc', dataNodes=[0.0.0.0]} [] at org.apache.cassandra.hadoop.ColumnFamilyInputFormat$SplitCallable.call(ColumnFamilyInputFormat.java:210)
{noformat}
In this split there is in fact ~40 million rows, and with a split size of 393216, it is expected ~100 splits to be returned.
I'm also very confused by the {{dataNodes=[0.0.0.0]}}. this looks to be wrong, i know that data lies on the third node: cassandra03, where-else the job is running on the first node: cassandra01.
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, fullscan-example1.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mck SembWever updated CASSANDRA-3150:
-------------------------------------
Affects Version/s: 0.8.5
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102126#comment-13102126 ]
Jonathan Ellis commented on CASSANDRA-3150:
-------------------------------------------
I don't know. I think what I would do would be to use sstablekeys to double-check how many rows there really are in the given split range.
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099263#comment-13099263 ]
Jonathan Ellis commented on CASSANDRA-3150:
-------------------------------------------
If Token's Comparable implementation is broken, anything is possible. But I don't think it is. In your case, for instance, OPP is sorting the shorter token correctly since 3 < 7.
Load won't decrease until you run cleanup.
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114115#comment-13114115 ]
Jonathan Ellis commented on CASSANDRA-3150:
-------------------------------------------
I'd love to think we fixed it but nothing in 0.8.6 changes looks relevant. I suspect it's still lurking to bite you.
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114220#comment-13114220 ]
Mck SembWever edited comment on CASSANDRA-3150 at 9/25/11 11:23 AM:
--------------------------------------------------------------------
Here's debug from a "full scan" job. It scans data over a full year (and since the cluster's ring range only holds 3 months of data such a job guarantees a full scan).
In the debug you see the splits.
{{`nodetool ring`}} gives
{noformat}Address DC Rack Status State Load Owns Token
Token(bytes[55555555555555555555555555555554])
152.90.241.22 DC1 RAC1 Up Normal 16.65 GB 33.33% Token(bytes[00])
152.90.241.23 DC2 RAC1 Up Normal 63.22 GB 33.33% Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa])
152.90.241.24 DC1 RAC1 Up Normal 72.4 KB 33.33% Token(bytes[55555555555555555555555555555554])
{noformat}
The problematic split ends up being
{noformat}ColumnFamilySplit{startToken='0528cbe0b2b5ff6b816c68b59973bcbc', endToken='2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', dataNodes=[cassandra02.finn.no]}{noformat}
was (Author: michaelsembwever):
Here's debug from a "full scan" job. It scans data over a full year (and since the cluster's ring range only hold 3 months of data this job guarantees a full scan).
In the debug you see the splits.
{{`nodetool ring`}} gives
{noformat}Address DC Rack Status State Load Owns Token
Token(bytes[55555555555555555555555555555554])
152.90.241.22 DC1 RAC1 Up Normal 16.65 GB 33.33% Token(bytes[00])
152.90.241.23 DC2 RAC1 Up Normal 63.22 GB 33.33% Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa])
152.90.241.24 DC1 RAC1 Up Normal 72.4 KB 33.33% Token(bytes[55555555555555555555555555555554])
{noformat}
The problematic split ends up being
{noformat}ColumnFamilySplit{startToken='0528cbe0b2b5ff6b816c68b59973bcbc', endToken='2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', dataNodes=[cassandra02.finn.no]}{noformat}
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, fullscan-example1.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099254#comment-13099254 ]
Mck SembWever edited comment on CASSANDRA-3150 at 9/7/11 7:43 PM:
------------------------------------------------------------------
What about the case where tokens of different length exist. Could get_range_slices be busted there?
I don't know if this is actually possible but from
{noformat}
Address Status State Load Owns Token
Token(bytes[76118303760208547436305468318170713656])
152.90.241.22 Up Normal 270.46 GB 33.33% Token(bytes[30303030303031333131313739353337303038d4e7f72db2ed11e09d7c68b59973a5d8])
152.90.241.24 Up Normal 247.89 GB 33.33% Token(bytes[303030303030313331323631393735313231381778518cc00711e0acb968b59973a5d8])
152.90.241.23 Up Normal 1.1 TB 33.33% Token(bytes[76118303760208547436305468318170713656])
{noformat}
you see the real tokens are very long compared to the initial_tokens the cluster was configured with. (The two long tokens have been moved off their initial_tokens, and to note the load on .23 never decreased to ~300GB as it should have...).
was (Author: michaelsembwever):
What about the case where tokens of different length exist.
I don't know if this is actually possible but from
{noformat}
Address Status State Load Owns Token
Token(bytes[76118303760208547436305468318170713656])
152.90.241.22 Up Normal 270.46 GB 33.33% Token(bytes[30303030303031333131313739353337303038d4e7f72db2ed11e09d7c68b59973a5d8])
152.90.241.24 Up Normal 247.89 GB 33.33% Token(bytes[303030303030313331323631393735313231381778518cc00711e0acb968b59973a5d8])
152.90.241.23 Up Normal 1.1 TB 33.33% Token(bytes[76118303760208547436305468318170713656])
{noformat}
you see the real tokens are very long compared to the initial_tokens the cluster was configured with. (The two long tokens have been moved off their initial_tokens, and to note the load on .23 never decreased to ~300GB as it should have...).
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mck SembWever updated CASSANDRA-3150:
-------------------------------------
Attachment: fullscan-example1.log
Here's debug from a "full scan" job. It scans data over a full year (and since the cluster's ring range only hold 3 months of data this job guarantees a full scan).
In the debug you see the splits.
{{`nodetool ring`}} gives
{noformat}Address DC Rack Status State Load Owns Token
Token(bytes[55555555555555555555555555555554])
152.90.241.22 DC1 RAC1 Up Normal 16.65 GB 33.33% Token(bytes[00])
152.90.241.23 DC2 RAC1 Up Normal 63.22 GB 33.33% Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa])
152.90.241.24 DC1 RAC1 Up Normal 72.4 KB 33.33% Token(bytes[55555555555555555555555555555554])
{noformat}
The problematic split ends up being
{noformat}ColumnFamilySplit{startToken='0528cbe0b2b5ff6b816c68b59973bcbc', endToken='2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', dataNodes=[cassandra02.finn.no]}{noformat}
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, fullscan-example1.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099254#comment-13099254 ]
Mck SembWever edited comment on CASSANDRA-3150 at 9/7/11 7:38 PM:
------------------------------------------------------------------
What about the case where tokens of different length exist.
I don't know if this is actually possible but from
{noformat}
Address Status State Load Owns Token
Token(bytes[76118303760208547436305468318170713656])
152.90.241.22 Up Normal 270.46 GB 33.33% Token(bytes[30303030303031333131313739353337303038d4e7f72db2ed11e09d7c68b59973a5d8])
152.90.241.24 Up Normal 247.89 GB 33.33% Token(bytes[303030303030313331323631393735313231381778518cc00711e0acb968b59973a5d8])
152.90.241.23 Up Normal 1.1 TB 33.33% Token(bytes[76118303760208547436305468318170713656])
{noformat}
you see the real tokens are very long compared to the initial_tokens the cluster was configured with. (The two long tokens have been moved off their initial_tokens, and to note the load on .23 never decreased to ~300GB as it should have...).
was (Author: michaelsembwever):
What about the case where tokens of different length exist.
I don't know if this is actually possible but from
{noformat}
Address Status State Load Owns Token
Token(bytes[76118303760208547436305468318170713656])
152.90.241.22 Up Normal 270.46 GB 33.33% Token(bytes[30303030303031333131313739353337303038d4e7f72db2ed11e09d7c68b59973a5d8])
152.90.241.24 Up Normal 247.89 GB 33.33% Token(bytes[303030303030313331323631393735313231381778518cc00711e0acb968b59973a5d8])
152.90.241.23 Up Normal 1.1 TB 33.33% Token(bytes[76118303760208547436305468318170713656])
{noformat}
you see the real tokens are very long compared to the initial_tokens the cluster was configured with. (The two long tokens has since been moved, and to note the load on .23 never decreased to ~300GB as it should have...).
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "Mck SembWever (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115726#comment-13115726 ]
Mck SembWever commented on CASSANDRA-3150:
------------------------------------------
bq. you haven't done any messing with index_interval by chance?
Once, around the time of this issue was created. I tried two smaller values (the first attempt gave OOM) but it didn't change much. The value is now at the original 128 again.
bq. So how many memtables do you have at once and how many rows can fit in a memtable?
Skinny rows. three columns: type, timestamp, data. type is a short string and indexed, timestamp is a long and indexed, data is a thrift serialised bean 100bytes to 1k.
memtables is set to 1024. (this was decided since we're using Xmx8g and didn't want huge number of sstables files).
So i guess that means we could fit up to ~5 million rows in a memtable...
But above i wrote that neither a flush nor a compact helped the situation... (in fact each hadoop job starts and finishes with a flush) at least it didn't with 0.8.4 and 0.8.5. Now it seems much better w/ 0.8.6. I see through the day tasks go over 100% but it doesn't exaggerate through the day as badly as it did before.
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, fullscan-example1.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102062#comment-13102062 ]
Mck SembWever edited comment on CASSANDRA-3150 at 9/10/11 6:02 PM:
-------------------------------------------------------------------
Debug from a task that was still running at 1200%
The initial split for this CFRR is
30303030303031333131313739353337303038d4e7f72db2ed11e09d7c68b59973a5d8 : 303030303030313331323631393735313231381778518cc00711e0acb968b59973a5d8
This job was run with
cassandra.input.split.size=196608
cassandra.range.batch.size=16000
therefore there shouldn't be more than 13 calls to get_range_slices(..) in this task. There was already 166 calls in this log.
What i can see here is that the original split for this task is just way too big and this comes from {{describe_splits(..)}}
which in turn depends on "index_interval". Reading {{StorageService.getSplits(..)}} i would guess that the split can in fact contain many more keys with the default sampling of 128. Question is how low can/should i bring index_interval ?
was (Author: michaelsembwever):
Debug from a task that was still running at 1200%
The initial split for this CFRR is
30303030303031333131313739353337303038d4e7f72db2ed11e09d7c68b59973a5d8 : 303030303030313331323631393735313231381778518cc00711e0acb968b59973a5d8
This job was run with
cassandra.input.split.size=196608
cassandra.range.batch.size=16000
therefore there shouldn't be more than 13 calls to get_range_slices(..) in this task. There was already 166 calls in this log.
What i can see here is that the original split for this task is just way too big and this comes from {{describe_splits(..)}}
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099269#comment-13099269 ]
Mck SembWever commented on CASSANDRA-3150:
------------------------------------------
bq. BOP is sorting the shorter token correctly since 3 < 7.
Sorry, so that doesn't explain this bug?
bq. Load won't decrease until you run cleanup.
Never worked.
repair and cleanup is run every night, the move was done one week ago and more than a couple of weeks ago.
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mck SembWever updated CASSANDRA-3150:
-------------------------------------
Attachment: attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
Debug from a task that was still running at 1200%
The initial split for this CFRR is
30303030303031333131313739353337303038d4e7f72db2ed11e09d7c68b59973a5d8 : 303030303030313331323631393735313231381778518cc00711e0acb968b59973a5d8
This job was run with
cassandra.input.split.size=196608
cassandra.range.batch.size=16000
therefore there shouldn't be more than 13 calls to get_range_slices(..) in this task. There was already 166 calls in this log.
What i can see here is that the original split for this task is just way too big and this comes from {{describe_splits(..)}}
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099234#comment-13099234 ]
Mck SembWever commented on CASSANDRA-3150:
------------------------------------------
Here keyRange is startToken to split.getEndToken()
startToken is updated each iterate to the last row read (each iterate is batchRowCount rows).
What happens is split.getEndToken() doesn't correspond to any of the rowKeys?
To me it reads that startToken will hop over split.getEndToken() and get_rage_slices(..) will start returning wrapping ranges. This will still return rows and so the iteration will continue, now forever.
The only way out for this code today is a) startToken equals split.getEndToken(), or b) get_range_slices(..) is called with startToken equals split.getEndToken() OR a gap so small there exists no rows in between.
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "Mck SembWever (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140229#comment-13140229 ]
Mck SembWever commented on CASSANDRA-3150:
------------------------------------------
As mentioned in CASSANDRA-3214 making the rpc listen only on one interface fixed the 0.0.0.0 problem.
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, fullscan-example1.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "T Jake Luciani (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115468#comment-13115468 ]
T Jake Luciani commented on CASSANDRA-3150:
-------------------------------------------
bq. This is the split that's receiving new data (5-10k rows/second).
So how many memtables do you have at once and how many rows can fit in a memtable? If you have large memtables and tiny rows that would throw getSplits() off since the splits are generated from SSTables only.
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, fullscan-example1.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102115#comment-13102115 ]
Mck SembWever commented on CASSANDRA-3150:
------------------------------------------
There's a lot to learn about cassandra so forgive my ignorance in so many areas.
So how can {{StorageService.getSplits(..)}} be so out of whack? Is there anything i can tune to better this situation?
(Or is there any other debug i can provide?)
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114220#comment-13114220 ]
Mck SembWever edited comment on CASSANDRA-3150 at 9/26/11 8:29 AM:
-------------------------------------------------------------------
{{fullscan-example1.log}} is debug from a "full scan" job. It scans data over a full year (and since the cluster's ring range only holds 3 months of data such a job guarantees a full scan).
In the debug you see the splits.
{{`nodetool ring`}} gives
{noformat}Address DC Rack Status State Load Owns Token
Token(bytes[55555555555555555555555555555554])
152.90.241.22 DC1 RAC1 Up Normal 16.65 GB 33.33% Token(bytes[00])
152.90.241.23 DC2 RAC1 Up Normal 63.22 GB 33.33% Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa])
152.90.241.24 DC1 RAC1 Up Normal 72.4 KB 33.33% Token(bytes[55555555555555555555555555555554])
{noformat}
The problematic split ends up being
{noformat}ColumnFamilySplit{startToken='0528cbe0b2b5ff6b816c68b59973bcbc', endToken='2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', dataNodes=[cassandra02.finn.no]}{noformat} This is the split that's receiving new data (5-10k rows/second). This new data is being written directly using {{StorageProxy.mutate(..)}} with code somewhat similar to the second example in [wiki: ScribeToCassandra|http://wiki.apache.org/cassandra/ScribeToCassandra].
This cluster has {{binary_memtable_throughput_in_mb: 1024}} and {{Xmx8g}}. There are 3 nodes in the cluster each 48g ram and 24cpus.
was (Author: michaelsembwever):
{{fullscan-example1.log}} is debug from a "full scan" job. It scans data over a full year (and since the cluster's ring range only holds 3 months of data such a job guarantees a full scan).
In the debug you see the splits.
{{`nodetool ring`}} gives
{noformat}Address DC Rack Status State Load Owns Token
Token(bytes[55555555555555555555555555555554])
152.90.241.22 DC1 RAC1 Up Normal 16.65 GB 33.33% Token(bytes[00])
152.90.241.23 DC2 RAC1 Up Normal 63.22 GB 33.33% Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa])
152.90.241.24 DC1 RAC1 Up Normal 72.4 KB 33.33% Token(bytes[55555555555555555555555555555554])
{noformat}
The problematic split ends up being
{noformat}ColumnFamilySplit{startToken='0528cbe0b2b5ff6b816c68b59973bcbc', endToken='2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', dataNodes=[cassandra02.finn.no]}{noformat} This is the split that's receiving new data (5-10k rows/second). This new data is being written directly using {{StorageProxy.mutate(..)}} with code somewhat similar to the second example in [wiki: ScribeToCassandra|http://wiki.apache.org/cassandra/ScribeToCassandra].
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, fullscan-example1.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "Mck SembWever (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115278#comment-13115278 ]
Mck SembWever commented on CASSANDRA-3150:
------------------------------------------
No. (the title is incorrect).
That split contains millions of rows, despite StorageService.getSplits(..) being called with splitSize 393216.
In the worse situations the split has 20 millions rows in it. This were all valid are had to be process for the hadoop job to produce accurate results.
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, fullscan-example1.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mck SembWever updated CASSANDRA-3150:
-------------------------------------
Attachment: CASSANDRA-3150.patch
If the split's end token does not match any of the row key tokens the RowIterator will never stop (see RowIterator:243)
This patch 1) presumes this is the problem, 2) compares each row token with the split end token and exits when need be (which only works on order preserving partitioners, and 3) stops iterating when totalRowCount has been read.
Just (3) has been tested and works.
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102109#comment-13102109 ]
Jonathan Ellis commented on CASSANDRA-3150:
-------------------------------------------
The index_interval dependency is this:
{code}
.
for (DecoratedKey sample : cfs.allKeySamples())
{
if (range.contains(sample.token))
keys.add(sample);
}
FBUtilities.sortSampledKeys(keys, range);
int splits = keys.size() * DatabaseDescriptor.getIndexInterval() / keysPerSplit;
{code}
so, we're taking the size of the key samples (1/index_interval) and multiplying it by index_interval to get an upper bound of the number of keys. So it doesn't matter what you do with your interval since it gets cancelled out.
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114220#comment-13114220 ]
Mck SembWever edited comment on CASSANDRA-3150 at 9/25/11 11:32 AM:
--------------------------------------------------------------------
{{fullscan-example1.log}} is debug from a "full scan" job. It scans data over a full year (and since the cluster's ring range only holds 3 months of data such a job guarantees a full scan).
In the debug you see the splits.
{{`nodetool ring`}} gives
{noformat}Address DC Rack Status State Load Owns Token
Token(bytes[55555555555555555555555555555554])
152.90.241.22 DC1 RAC1 Up Normal 16.65 GB 33.33% Token(bytes[00])
152.90.241.23 DC2 RAC1 Up Normal 63.22 GB 33.33% Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa])
152.90.241.24 DC1 RAC1 Up Normal 72.4 KB 33.33% Token(bytes[55555555555555555555555555555554])
{noformat}
The problematic split ends up being
{noformat}ColumnFamilySplit{startToken='0528cbe0b2b5ff6b816c68b59973bcbc', endToken='2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', dataNodes=[cassandra02.finn.no]}{noformat} This is the split that's receiving new data (5-10k rows/second). This new data is being written directly using {{StorageProxy.mutate(..)}} with code somewhat similar to the second example in [wiki: ScribeToCassandra|http://wiki.apache.org/cassandra/ScribeToCassandra].
was (Author: michaelsembwever):
{{fullscan-example1.log}} is debug from a "full scan" job. It scans data over a full year (and since the cluster's ring range only holds 3 months of data such a job guarantees a full scan).
In the debug you see the splits.
{{`nodetool ring`}} gives
{noformat}Address DC Rack Status State Load Owns Token
Token(bytes[55555555555555555555555555555554])
152.90.241.22 DC1 RAC1 Up Normal 16.65 GB 33.33% Token(bytes[00])
152.90.241.23 DC2 RAC1 Up Normal 63.22 GB 33.33% Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa])
152.90.241.24 DC1 RAC1 Up Normal 72.4 KB 33.33% Token(bytes[55555555555555555555555555555554])
{noformat}
The problematic split ends up being
{noformat}ColumnFamilySplit{startToken='0528cbe0b2b5ff6b816c68b59973bcbc', endToken='2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', dataNodes=[cassandra02.finn.no]}{noformat} This is the split that's receiving new data (5-10k rows/second).
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, fullscan-example1.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099247#comment-13099247 ]
Jonathan Ellis commented on CASSANDRA-3150:
-------------------------------------------
bq. a gap so small there exists no rows in between
Right. So you page through with startToken increasing, until either you hit endToken or you get no rows back.
(Recall that the rows from get_range_slices come back in token order.)
> ColumnFormatRecordReader loops forever
> --------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops
forever (StorageService.getSplits(..) out of whack)
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115451#comment-13115451 ]
Jonathan Ellis commented on CASSANDRA-3150:
-------------------------------------------
you haven't done any messing with index_interval by chance?
how much of the difference between 400K and 20M can be explained by new rows being added?
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, fullscan-example1.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113326#comment-13113326 ]
Mck SembWever edited comment on CASSANDRA-3150 at 9/23/11 7:30 PM:
-------------------------------------------------------------------
This is persisting to be a problem. And continuously gets worse through the day.
I can see two thirds of my cf being read from one split.
It doesn't matter if it's ByteOrderingPartition or RandomPartitioner. Although i'm having a greater problem with the former. A compact and restart didn't help.
I've attached two screenshots from hadoop webpages (you see 36million map input records when cassandra.input.split.size is set to 393216).
bq. ... double-check how many rows there really are in the given split range.
I can confirm that if i let these jobs run through to their completion (despite how terribly long they may take) that the results are correct. It would seem that the split ranges are incorrect (not the rows within in them).
was (Author: michaelsembwever):
This is persisting to be a problem. And continuously gets worse through the day.
I can see like half my cf being read from one split.
It doesn't matter if it's ByteOrderingPartition or RandomPartitioner.
I've attached two screenshots from hadoop webpages (you see 36million map input records when cassandra.input.split.size is set to 393216).
bq. ... double-check how many rows there really are in the given split range.
I can confirm that if i let these jobs run through to their completion (despite how terribly long they may take) that the results are correct. It would seem that the split ranges are incorrect (not the rows within in them).
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150)
ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
whack)
Posted by "Mck SembWever (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114220#comment-13114220 ]
Mck SembWever edited comment on CASSANDRA-3150 at 9/25/11 11:30 AM:
--------------------------------------------------------------------
{{fullscan-example1.log}} is debug from a "full scan" job. It scans data over a full year (and since the cluster's ring range only holds 3 months of data such a job guarantees a full scan).
In the debug you see the splits.
{{`nodetool ring`}} gives
{noformat}Address DC Rack Status State Load Owns Token
Token(bytes[55555555555555555555555555555554])
152.90.241.22 DC1 RAC1 Up Normal 16.65 GB 33.33% Token(bytes[00])
152.90.241.23 DC2 RAC1 Up Normal 63.22 GB 33.33% Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa])
152.90.241.24 DC1 RAC1 Up Normal 72.4 KB 33.33% Token(bytes[55555555555555555555555555555554])
{noformat}
The problematic split ends up being
{noformat}ColumnFamilySplit{startToken='0528cbe0b2b5ff6b816c68b59973bcbc', endToken='2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', dataNodes=[cassandra02.finn.no]}{noformat} This is the split that's receiving new data (5-10k rows/second).
was (Author: michaelsembwever):
Here's debug from a "full scan" job. It scans data over a full year (and since the cluster's ring range only holds 3 months of data such a job guarantees a full scan).
In the debug you see the splits.
{{`nodetool ring`}} gives
{noformat}Address DC Rack Status State Load Owns Token
Token(bytes[55555555555555555555555555555554])
152.90.241.22 DC1 RAC1 Up Normal 16.65 GB 33.33% Token(bytes[00])
152.90.241.23 DC2 RAC1 Up Normal 63.22 GB 33.33% Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa])
152.90.241.24 DC1 RAC1 Up Normal 72.4 KB 33.33% Token(bytes[55555555555555555555555555555554])
{noformat}
The problematic split ends up being
{noformat}ColumnFamilySplit{startToken='0528cbe0b2b5ff6b816c68b59973bcbc', endToken='2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', dataNodes=[cassandra02.finn.no]}{noformat} This is the split that's receiving new data (5-10k rows/second).
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Mck SembWever
> Assignee: Mck SembWever
> Priority: Critical
> Fix For: 0.8.6
>
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for task_201109212019_1060_m_000029 - Mozilla Firefox.png, Screenshot-Hadoop map task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png, attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log, fullscan-example1.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira