You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Patrik Modesto <pa...@gmail.com> on 2011/08/02 16:22:45 UTC
RF=1
Hi all!
I've a test cluster of 4 nodes running cassandra 0.7.8, with one
keyspace with RF=1, each node owns 25% of the data. As long as all
nodes are alive, there is no problem, but when I shut down just one
node I get UnavailableException in my application. cassandra-cli
returns "null" and hadoop mapreduce task won't start at all.
Loosing one node is not a problem for me, the data are not important,
loosing even half the cluster is not a problem as long as everything
runs just as with a full cluster.
The error from hadoop is like this:
Exception in thread "main" java.io.IOException: Could not get input splits
at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:120)
at cz.xxx.yyy.zzz.DelegatingInputFormat.getSplits(DelegatingInputFormat.java:111)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961)
at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
at cz.xxx.yyy.zzz.ContextIndexer.run(ContextIndexer.java:663)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at cz.xxx.yyy.zzz.ContextIndexer.main(ContextIndexer.java:94)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Caused by: java.util.concurrent.ExecutionException:
java.io.IOException: failed connecting to all endpoints 10.0.18.87
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:116)
... 20 more
Caused by: java.io.IOException: failed connecting to all endpoints 10.0.18.87
at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSubSplits(ColumnFamilyInputFormat.java:197)
at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.access$200(ColumnFamilyInputFormat.java:67)
at org.apache.cassandra.hadoop.ColumnFamilyInputFormat$SplitCallable.call(ColumnFamilyInputFormat.java:153)
at org.apache.cassandra.hadoop.ColumnFamilyInputFormat$SplitCallable.call(ColumnFamilyInputFormat.java:138)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Re: RF=1
Posted by Paul Loy <ke...@gmail.com>.
this is expected behaviour. Either increase RF or do a nodetool decommission
on a node to remove it from the ring.
On Tue, Aug 2, 2011 at 3:22 PM, Patrik Modesto <pa...@gmail.com>wrote:
> Hi all!
>
> I've a test cluster of 4 nodes running cassandra 0.7.8, with one
> keyspace with RF=1, each node owns 25% of the data. As long as all
> nodes are alive, there is no problem, but when I shut down just one
> node I get UnavailableException in my application. cassandra-cli
> returns "null" and hadoop mapreduce task won't start at all.
>
> Loosing one node is not a problem for me, the data are not important,
> loosing even half the cluster is not a problem as long as everything
> runs just as with a full cluster.
>
> The error from hadoop is like this:
> Exception in thread "main" java.io.IOException: Could not get input splits
> at
> org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:120)
> at
> cz.xxx.yyy.zzz.DelegatingInputFormat.getSplits(DelegatingInputFormat.java:111)
> at
> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944)
> at
> org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961)
> at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
> at cz.xxx.yyy.zzz.ContextIndexer.run(ContextIndexer.java:663)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at cz.xxx.yyy.zzz.ContextIndexer.main(ContextIndexer.java:94)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: java.util.concurrent.ExecutionException:
> java.io.IOException: failed connecting to all endpoints 10.0.18.87
> at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:116)
> ... 20 more
> Caused by: java.io.IOException: failed connecting to all endpoints
> 10.0.18.87
> at
> org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSubSplits(ColumnFamilyInputFormat.java:197)
> at
> org.apache.cassandra.hadoop.ColumnFamilyInputFormat.access$200(ColumnFamilyInputFormat.java:67)
> at
> org.apache.cassandra.hadoop.ColumnFamilyInputFormat$SplitCallable.call(ColumnFamilyInputFormat.java:153)
> at
> org.apache.cassandra.hadoop.ColumnFamilyInputFormat$SplitCallable.call(ColumnFamilyInputFormat.java:138)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
--
---------------------------------------------
Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy
Re: RF=1
Posted by Patrik Modesto <pa...@gmail.com>.
And one more patch: http://pastebin.com/zfNPjtQz
This one handles a case where there are no nodes available for a
slice. For example where the is a keyspace with RF=1 and a node is
shut down. Its range of keys gets ignored.
Regards,
P.
On Wed, Aug 17, 2011 at 13:28, Patrik Modesto <pa...@gmail.com> wrote:
> Hi,
>
> while I was investigating this issue, I've found that hadoop+cassandra
> don't work if you stop even just one node in the cluster. It doesn't
> depend on RF. ColumnFamilyRecordReader gets list of nodes (acording
> the RF) but chooses just the local host and if there is no cassandra
> running localy it throws RuntimeError exception. Which in turn marks
> the MapReduce task as failed.
>
> I've created a patch that makes ColumnFamilyRecordReader to try the
> local node and if it fails tries the other nodes in it's list. The
> patch is here http://pastebin.com/0RdQ0HMx I think attachements are
> not allowed on this ML.
>
> Please test it and apply. It's for 0.7.8 version.
>
> Regards,
> P.
>
>
> On Wed, Aug 3, 2011 at 13:59, aaron morton <aa...@thelastpickle.com> wrote:
>> If you want to take a look o.a.c.hadoop.ColumnFamilyRecordReader.getSplits() is the function that gets the splits.
>>
>>
>> Cheers
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 3 Aug 2011, at 16:18, Patrik Modesto wrote:
>>
>>> On Tue, Aug 2, 2011 at 23:10, Jeremiah Jordan
>>> <je...@morningstar.com> wrote:
>>>> If you have RF=1, taking one node down is going to cause 25% of your
>>>> data to be unavailable. If you want to tolerate a machines going down
>>>> you need to have at least RF=2, if you want to use quorum and have a
>>>> machine go down, you need at least RF=3.
>>>
>>> I know I can have RF > 1 but I have limited resources and I don't care
>>> lossing 25% of the data. RF > 1 basicaly means if a node goes down I
>>> have the data elsewhere, but what I need is if node goes down just
>>> ignore its range. I can handle it in my applications using thrift, but
>>> the hadoop-mapreduce can't handle it. It just fails with "Exception in
>>> thread "main" java.io.IOException: Could not get input splits". Is
>>> there a way to say ignore this range to hadoop?
>>>
>>> Regards,
>>> P.
>>
>>
>
Re: RF=1 w/ hadoop jobs
Posted by Mick Semb Wever <mc...@apache.org>.
On Mon, 2011-09-05 at 21:52 +0200, Patrik Modesto wrote:
> I'm not sure about 0.8.x and 0.7.9 (to be released today with your
> patch) but 0.7.8 will fail even with RF>1 when there is Hadoop
> TaskTracer without local Cassandra. So increasing RF is not a
> solution.
This isn't true (or not the intention).
If you increase RF then yes the task will fail but it will get re-run on
the next replica. So the job takes longer but should still work.
~mck
--
"This is my simple religion. There is no need for temples; no need for
complicated philosophy. Our own brain, our own heart is our temple; the
philosophy is kindness." The Dalai Lama
| http://semb.wever.org | http://sesat.no |
| http://tech.finn.no | Java XSS Filter |
Re: RF=1 w/ hadoop jobs
Posted by Patrik Modesto <pa...@gmail.com>.
On Mon, Sep 5, 2011 at 09:39, Mick Semb Wever <mc...@apache.org> wrote:
> I've entered a jira issue covering this request.
> https://issues.apache.org/jira/browse/CASSANDRA-3136
>
> Would you mind attaching your patch to the issue.
> (No review of it will happen anywhere else.)
I see Jonathan didn't change his mind, as the ticket was resolved
"won't fix". So that's it. I'm not going to attach the patch until
another core Cassandra developer is interested in the use-cases
mentioned there.
I'm not sure about 0.8.x and 0.7.9 (to be released today with your
patch) but 0.7.8 will fail even with RF>1 when there is Hadoop
TaskTracer without local Cassandra. So increasing RF is not a
solution.
Regards,
Patrik
Re: RF=1 w/ hadoop jobs
Posted by Mick Semb Wever <mc...@apache.org>.
On Fri, 2011-09-02 at 09:28 +0200, Patrik Modesto wrote:
> We use Cassandra as a storage for web-pages, we store the HTML, all
> URLs that has the same HTML data and some computed data. We run Hadoop
> MR jobs to compute lexical and thematical data for each page and for
> exporting the data to a binary files for later use. URL gets to a
> Cassandra on user request (a pageview) so if we delete an URL, it gets
> back quickly if the page is active. Because of that and because there
> is lots of data, we have the keyspace set to RF=1. We can drop the
> whole keyspace and it will regenerate quickly and would contain only
> fresh data, so we don't care about lossing a node.
I've entered a jira issue covering this request.
https://issues.apache.org/jira/browse/CASSANDRA-3136
Would you mind attaching your patch to the issue.
(No review of it will happen anywhere else.)
~mck
--
“Innovators and creative geniuses cannot be reared in schools. They are
precisely the men who defy what the school has taught them.” - Ludwig
von Mises
| http://semb.wever.org | http://sesat.no |
| http://tech.finn.no | Java XSS Filter |
Re: RF=1 w/ hadoop jobs
Posted by Patrik Modesto <pa...@gmail.com>.
On Fri, Sep 2, 2011 at 08:54, Mick Semb Wever <mc...@apache.org> wrote:
> Patrik: is it possible to describe the use-case you have here?
Sure.
We use Cassandra as a storage for web-pages, we store the HTML, all
URLs that has the same HTML data and some computed data. We run Hadoop
MR jobs to compute lexical and thematical data for each page and for
exporting the data to a binary files for later use. URL gets to a
Cassandra on user request (a pageview) so if we delete an URL, it gets
back quickly if the page is active. Because of that and because there
is lots of data, we have the keyspace set to RF=1. We can drop the
whole keyspace and it will regenerate quickly and would contain only
fresh data, so we don't care about lossing a node. But Hadoop does
care, well to be specific the Cassnadra ColumnInputFormat and
ColumnRecortReader are the problem parts. If I stop one Cassandra node
all MR jobs that read/write Cassandra fail. In our case, it doesn't
matter, we can skip the range of URLs. The MR jobs run in a tight
loop, so when the node is back with it's data, we use them. It's not
only about some HW crash but it makes maintenance quite difficult. To
stop a Cassandra node, you have to stop tasktracker there too which is
unfortunate as there are another MR jobs that don't need Cassandra and
can happily run.
Regards,
P.
Re: RF=1 w/ hadoop jobs
Posted by Mick Semb Wever <mc...@apache.org>.
On Fri, 2011-09-02 at 08:20 +0200, Patrik Modesto wrote:
> As Jonathan
> already explained himself: "ignoring unavailable ranges is a
> misfeature, imo"
Generally it's not what one would want i think.
But I can see the case when data is to be treated volatile and ignoring
unavailable ranges may be acceptable.
For example if you searching for something or some-pattern and one hit
is enough. If you get the hit it's a positive result regardless if
ranges were ignored, if you don't and you *know* there was a range
ignored along the way you can re-run the job later. The worse case
scenario here is no worse than the job always failing on you. Although
some indication of ranges ignored is required.
Another example is when your just trying to extract a small random
sample (like a pig SAMPLE) of data out of cassandra.
Patrik: is it possible to describe the use-case you have here?
~mck
--
“The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore, all
progress depends on the unreasonable man.” - George Bernard Shaw
| http://semb.wever.org | http://sesat.no |
| http://tech.finn.no | Java XSS Filter |
Re: RF=1 w/ hadoop jobs
Posted by Patrik Modesto <pa...@gmail.com>.
Hi,
On Thu, Sep 1, 2011 at 12:36, Mck <mc...@apache.org> wrote:
>> It's available here: http://pastebin.com/hhrr8m9P (for version 0.7.8)
>
> I'm interested in this patch and see it's usefulness but no one will act
> until you attach it to an issue. (I think a new issue is appropriate
> here).
I'm glad someone is interestet in my patch usefull. As Jonathan
already explained himself: "ignoring unavailable ranges is a
misfeature, imo" I'm thinking opening a new ticket without support
from more users is useless ATM. Please test the patch and if you like
it, than there is time for ticket.
Regards,
P.
Re: RF=1 w/ hadoop jobs
Posted by Mck <mc...@apache.org>.
On Thu, 2011-08-18 at 08:54 +0200, Patrik Modesto wrote:
> But there is the another problem with Hadoop-Cassandra, if there is no
> node available for a range of keys, it fails on RuntimeError. For
> example having a keyspace with RF=1 and a node is down all MapReduce
> tasks fail.
CASSANDRA-2388 is related but not the same.
Before 0.8.4 the behaviour was if the local cassandra node didn't have
the split's data the tasktracker would connect to another cassandra node
where the split's data could be found.
So even <0.8.4 with RF=1 you would have your hadoop job fail.
Although I've reopened CASSANDRA-2388 (and reverted the code locally)
because the new behaviour in 0.8.4 leads to abysmal tasktracker
throughput (for me task allocation doesn't seem to honour data-locality
according to split.getLocations()).
> I've reworked my previous patch, that was addressing this
> issue and now there are ConfigHelper methods for enable/disable
> ignoring unavailable ranges.
> It's available here: http://pastebin.com/hhrr8m9P (for version 0.7.8)
I'm interested in this patch and see it's usefulness but no one will act
until you attach it to an issue. (I think a new issue is appropriate
here).
~mck
Re: RF=1
Posted by Jonathan Ellis <jb...@gmail.com>.
(a) this really isn't the right forum to review patches; I've pointed
out the relevant jira ticket
(b) ignoring unavailable ranges is a misfeature, imo
On Fri, Aug 19, 2011 at 8:11 AM, Patrik Modesto
<pa...@gmail.com> wrote:
> Is there really no interest in the patch?
>
> P.
>
> On Thu, Aug 18, 2011 at 08:54, Patrik Modesto <pa...@gmail.com> wrote:
>> On Wed, Aug 17, 2011 at 17:08, Jonathan Ellis <jb...@gmail.com> wrote:
>>> See https://issues.apache.org/jira/browse/CASSANDRA-2388
>>
>> Ok, thanks for the JIRA ticker. I've found that very same problem
>> during my work on ignoring unavailabla ranges.
>>
>> But there is the another problem with Hadoop-Cassandra, if there is no
>> node available for a range of keys, it fails on RuntimeError. For
>> example having a keyspace with RF=1 and a node is down all MapReduce
>> tasks fail. I've reworked my previous patch, that was addressing this
>> issue and now there are ConfigHelper methods for enable/disable
>> ignoring unavailable ranges.
>> It's available here: http://pastebin.com/hhrr8m9P (for version 0.7.8)
>> I hope it can be applied.
>>
>> Regards,
>> Patrik
>>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: RF=1
Posted by Patrik Modesto <pa...@gmail.com>.
Is there really no interest in the patch?
P.
On Thu, Aug 18, 2011 at 08:54, Patrik Modesto <pa...@gmail.com> wrote:
> On Wed, Aug 17, 2011 at 17:08, Jonathan Ellis <jb...@gmail.com> wrote:
>> See https://issues.apache.org/jira/browse/CASSANDRA-2388
>
> Ok, thanks for the JIRA ticker. I've found that very same problem
> during my work on ignoring unavailabla ranges.
>
> But there is the another problem with Hadoop-Cassandra, if there is no
> node available for a range of keys, it fails on RuntimeError. For
> example having a keyspace with RF=1 and a node is down all MapReduce
> tasks fail. I've reworked my previous patch, that was addressing this
> issue and now there are ConfigHelper methods for enable/disable
> ignoring unavailable ranges.
> It's available here: http://pastebin.com/hhrr8m9P (for version 0.7.8)
> I hope it can be applied.
>
> Regards,
> Patrik
>
Re: RF=1
Posted by Patrik Modesto <pa...@gmail.com>.
On Wed, Aug 17, 2011 at 17:08, Jonathan Ellis <jb...@gmail.com> wrote:
> See https://issues.apache.org/jira/browse/CASSANDRA-2388
Ok, thanks for the JIRA ticker. I've found that very same problem
during my work on ignoring unavailabla ranges.
But there is the another problem with Hadoop-Cassandra, if there is no
node available for a range of keys, it fails on RuntimeError. For
example having a keyspace with RF=1 and a node is down all MapReduce
tasks fail. I've reworked my previous patch, that was addressing this
issue and now there are ConfigHelper methods for enable/disable
ignoring unavailable ranges.
It's available here: http://pastebin.com/hhrr8m9P (for version 0.7.8)
I hope it can be applied.
Regards,
Patrik
Re: RF=1
Posted by Jonathan Ellis <jb...@gmail.com>.
See https://issues.apache.org/jira/browse/CASSANDRA-2388
On Wed, Aug 17, 2011 at 6:28 AM, Patrik Modesto
<pa...@gmail.com> wrote:
> Hi,
>
> while I was investigating this issue, I've found that hadoop+cassandra
> don't work if you stop even just one node in the cluster. It doesn't
> depend on RF. ColumnFamilyRecordReader gets list of nodes (acording
> the RF) but chooses just the local host and if there is no cassandra
> running localy it throws RuntimeError exception. Which in turn marks
> the MapReduce task as failed.
>
> I've created a patch that makes ColumnFamilyRecordReader to try the
> local node and if it fails tries the other nodes in it's list. The
> patch is here http://pastebin.com/0RdQ0HMx I think attachements are
> not allowed on this ML.
>
> Please test it and apply. It's for 0.7.8 version.
>
> Regards,
> P.
>
>
> On Wed, Aug 3, 2011 at 13:59, aaron morton <aa...@thelastpickle.com> wrote:
>> If you want to take a look o.a.c.hadoop.ColumnFamilyRecordReader.getSplits() is the function that gets the splits.
>>
>>
>> Cheers
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 3 Aug 2011, at 16:18, Patrik Modesto wrote:
>>
>>> On Tue, Aug 2, 2011 at 23:10, Jeremiah Jordan
>>> <je...@morningstar.com> wrote:
>>>> If you have RF=1, taking one node down is going to cause 25% of your
>>>> data to be unavailable. If you want to tolerate a machines going down
>>>> you need to have at least RF=2, if you want to use quorum and have a
>>>> machine go down, you need at least RF=3.
>>>
>>> I know I can have RF > 1 but I have limited resources and I don't care
>>> lossing 25% of the data. RF > 1 basicaly means if a node goes down I
>>> have the data elsewhere, but what I need is if node goes down just
>>> ignore its range. I can handle it in my applications using thrift, but
>>> the hadoop-mapreduce can't handle it. It just fails with "Exception in
>>> thread "main" java.io.IOException: Could not get input splits". Is
>>> there a way to say ignore this range to hadoop?
>>>
>>> Regards,
>>> P.
>>
>>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: RF=1
Posted by Patrik Modesto <pa...@gmail.com>.
Hi,
while I was investigating this issue, I've found that hadoop+cassandra
don't work if you stop even just one node in the cluster. It doesn't
depend on RF. ColumnFamilyRecordReader gets list of nodes (acording
the RF) but chooses just the local host and if there is no cassandra
running localy it throws RuntimeError exception. Which in turn marks
the MapReduce task as failed.
I've created a patch that makes ColumnFamilyRecordReader to try the
local node and if it fails tries the other nodes in it's list. The
patch is here http://pastebin.com/0RdQ0HMx I think attachements are
not allowed on this ML.
Please test it and apply. It's for 0.7.8 version.
Regards,
P.
On Wed, Aug 3, 2011 at 13:59, aaron morton <aa...@thelastpickle.com> wrote:
> If you want to take a look o.a.c.hadoop.ColumnFamilyRecordReader.getSplits() is the function that gets the splits.
>
>
> Cheers
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 3 Aug 2011, at 16:18, Patrik Modesto wrote:
>
>> On Tue, Aug 2, 2011 at 23:10, Jeremiah Jordan
>> <je...@morningstar.com> wrote:
>>> If you have RF=1, taking one node down is going to cause 25% of your
>>> data to be unavailable. If you want to tolerate a machines going down
>>> you need to have at least RF=2, if you want to use quorum and have a
>>> machine go down, you need at least RF=3.
>>
>> I know I can have RF > 1 but I have limited resources and I don't care
>> lossing 25% of the data. RF > 1 basicaly means if a node goes down I
>> have the data elsewhere, but what I need is if node goes down just
>> ignore its range. I can handle it in my applications using thrift, but
>> the hadoop-mapreduce can't handle it. It just fails with "Exception in
>> thread "main" java.io.IOException: Could not get input splits". Is
>> there a way to say ignore this range to hadoop?
>>
>> Regards,
>> P.
>
>
Re: RF=1
Posted by aaron morton <aa...@thelastpickle.com>.
If you want to take a look o.a.c.hadoop.ColumnFamilyRecordReader.getSplits() is the function that gets the splits.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 3 Aug 2011, at 16:18, Patrik Modesto wrote:
> On Tue, Aug 2, 2011 at 23:10, Jeremiah Jordan
> <je...@morningstar.com> wrote:
>> If you have RF=1, taking one node down is going to cause 25% of your
>> data to be unavailable. If you want to tolerate a machines going down
>> you need to have at least RF=2, if you want to use quorum and have a
>> machine go down, you need at least RF=3.
>
> I know I can have RF > 1 but I have limited resources and I don't care
> lossing 25% of the data. RF > 1 basicaly means if a node goes down I
> have the data elsewhere, but what I need is if node goes down just
> ignore its range. I can handle it in my applications using thrift, but
> the hadoop-mapreduce can't handle it. It just fails with "Exception in
> thread "main" java.io.IOException: Could not get input splits". Is
> there a way to say ignore this range to hadoop?
>
> Regards,
> P.
Re: RF=1
Posted by Patrik Modesto <pa...@gmail.com>.
On Tue, Aug 2, 2011 at 23:10, Jeremiah Jordan
<je...@morningstar.com> wrote:
> If you have RF=1, taking one node down is going to cause 25% of your
> data to be unavailable. If you want to tolerate a machines going down
> you need to have at least RF=2, if you want to use quorum and have a
> machine go down, you need at least RF=3.
I know I can have RF > 1 but I have limited resources and I don't care
lossing 25% of the data. RF > 1 basicaly means if a node goes down I
have the data elsewhere, but what I need is if node goes down just
ignore its range. I can handle it in my applications using thrift, but
the hadoop-mapreduce can't handle it. It just fails with "Exception in
thread "main" java.io.IOException: Could not get input splits". Is
there a way to say ignore this range to hadoop?
Regards,
P.
Re: RF=1
Posted by Jeremiah Jordan <je...@morningstar.com>.
If you have RF=1, taking one node down is going to cause 25% of your
data to be unavailable. If you want to tolerate a machines going down
you need to have at least RF=2, if you want to use quorum and have a
machine go down, you need at least RF=3.
On Tue, 2011-08-02 at 16:22 +0200, Patrik Modesto wrote:
> Hi all!
>
> I've a test cluster of 4 nodes running cassandra 0.7.8, with one
> keyspace with RF=1, each node owns 25% of the data. As long as all
> nodes are alive, there is no problem, but when I shut down just one
> node I get UnavailableException in my application. cassandra-cli
> returns "null" and hadoop mapreduce task won't start at all.
>
> Loosing one node is not a problem for me, the data are not important,
> loosing even half the cluster is not a problem as long as everything
> runs just as with a full cluster.
>
> The error from hadoop is like this:
> Exception in thread "main" java.io.IOException: Could not get input splits
> at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:120)
> at cz.xxx.yyy.zzz.DelegatingInputFormat.getSplits(DelegatingInputFormat.java:111)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944)
> at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961)
> at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
> at cz.xxx.yyy.zzz.ContextIndexer.run(ContextIndexer.java:663)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at cz.xxx.yyy.zzz.ContextIndexer.main(ContextIndexer.java:94)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: java.util.concurrent.ExecutionException:
> java.io.IOException: failed connecting to all endpoints 10.0.18.87
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:116)
> ... 20 more
> Caused by: java.io.IOException: failed connecting to all endpoints 10.0.18.87
> at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSubSplits(ColumnFamilyInputFormat.java:197)
> at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.access$200(ColumnFamilyInputFormat.java:67)
> at org.apache.cassandra.hadoop.ColumnFamilyInputFormat$SplitCallable.call(ColumnFamilyInputFormat.java:153)
> at org.apache.cassandra.hadoop.ColumnFamilyInputFormat$SplitCallable.call(ColumnFamilyInputFormat.java:138)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)