You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Joost Ouwerkerk (JIRA)" <ji...@apache.org> on 2010/05/16 21:50:42 UTC
[jira] Created: (CASSANDRA-1096) Sequential splits causing
unbalanced MapReduce load
Sequential splits causing unbalanced MapReduce load
---------------------------------------------------
Key: CASSANDRA-1096
URL: https://issues.apache.org/jira/browse/CASSANDRA-1096
Project: Cassandra
Issue Type: Bug
Components: Hadoop
Affects Versions: 0.6
Reporter: Joost Ouwerkerk
Since CassandraInputFormat returns an ordered list of splits, when there are many splits (e.g. hundreds or more) the load on cassandra is horribly unbalanced. e.g. if I have 30 tasks processing 600 splits, then the rows for the first 30 splits are all located on the same one or two nodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-1096) Sequential splits causing
unbalanced MapReduce load
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-1096.
---------------------------------------
Assignee: Joost Ouwerkerk
Resolution: Fixed
committed. nice troubleshooting work!
> Sequential splits causing unbalanced MapReduce load
> ---------------------------------------------------
>
> Key: CASSANDRA-1096
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1096
> Project: Cassandra
> Issue Type: Improvement
> Components: Hadoop
> Affects Versions: 0.6
> Reporter: Joost Ouwerkerk
> Assignee: Joost Ouwerkerk
> Priority: Minor
> Fix For: 0.6.2
>
> Attachments: CASSANDRA-1096.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Since CassandraInputFormat returns an ordered list of splits, when there are many splits (e.g. hundreds or more) the load on cassandra is horribly unbalanced. e.g. if I have 30 tasks processing 600 splits, then the rows for the first 30 splits are all located on the same one or two nodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1096) Sequential splits causing
unbalanced MapReduce load
Posted by "Joost Ouwerkerk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joost Ouwerkerk updated CASSANDRA-1096:
---------------------------------------
Attachment: CASSANDRA-1096.patch
I added Collections.shuffle(splits) before returning the splits in getSplits(). As a result, the load is much better distributed, throughput was increased (about 3X in my case) and TimedOutExceptions were all but eliminated.
> Sequential splits causing unbalanced MapReduce load
> ---------------------------------------------------
>
> Key: CASSANDRA-1096
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1096
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 0.6
> Reporter: Joost Ouwerkerk
> Attachments: CASSANDRA-1096.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Since CassandraInputFormat returns an ordered list of splits, when there are many splits (e.g. hundreds or more) the load on cassandra is horribly unbalanced. e.g. if I have 30 tasks processing 600 splits, then the rows for the first 30 splits are all located on the same one or two nodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1096) Sequential splits causing
unbalanced MapReduce load
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1096:
--------------------------------------
Issue Type: Improvement (was: Bug)
Fix Version/s: 0.6.2
Priority: Minor (was: Major)
> Sequential splits causing unbalanced MapReduce load
> ---------------------------------------------------
>
> Key: CASSANDRA-1096
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1096
> Project: Cassandra
> Issue Type: Improvement
> Components: Hadoop
> Affects Versions: 0.6
> Reporter: Joost Ouwerkerk
> Priority: Minor
> Fix For: 0.6.2
>
> Attachments: CASSANDRA-1096.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Since CassandraInputFormat returns an ordered list of splits, when there are many splits (e.g. hundreds or more) the load on cassandra is horribly unbalanced. e.g. if I have 30 tasks processing 600 splits, then the rows for the first 30 splits are all located on the same one or two nodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.