You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@gora.apache.org by kaveh minooie <ka...@plutoz.com> on 2013/08/29 02:48:41 UTC
too many HTimedOutException in reduce tasks
Hi every one,
this is an example of what I am getting :
2013-08-28 16:41:09,520 INFO org.apache.gora.mapreduce.GoraRecordWriter:
Flushing the datastore after 10000 records
2013-08-28 16:41:19,556 WARN
me.prettyprint.cassandra.connection.HConnectionManager: Could not
fullfill request on this host CassandraClient<crawler1:9160-2>
2013-08-28 16:41:19,557 WARN
me.prettyprint.cassandra.connection.HConnectionManager: Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
at me.prettyprint.cassandra.model.MutatorImpl.insert(MutatorImpl.java:69)
at
org.apache.gora.cassandra.store.HectorUtils.insertColumn(HectorUtils.java:52)
at
org.apache.gora.cassandra.store.CassandraClient.addColumn(CassandraClient.java:197)
at
org.apache.gora.cassandra.store.CassandraStore.addOrUpdateField(CassandraStore.java:353)
at
org.apache.gora.cassandra.store.CassandraStore.flush(CassandraStore.java:228)
at
org.apache.gora.mapreduce.GoraRecordWriter.write(GoraRecordWriter.java:70)
at
org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:586)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.nutch.crawl.DbUpdateReducer.reduce(DbUpdateReducer.java:202)
at org.apache.nutch.crawl.DbUpdateReducer.reduce(DbUpdateReducer.java:39)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
at
org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
at
me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
at
me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
at
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
... 20 more
this is happening in virtually all of my reduce tasks, and I think as a
result of this, I get this for about 15% of my reduce tasks:
Task attempt_201308151359_0006_r_000014_0 failed to report status for
1800 seconds. Killing!
is there anything that I can do to improve this situation? maybe
increasing the timeout time some where?
thanks,
--
Kaveh Minooie
Re: too many HTimedOutException in reduce tasks
Posted by kaveh minooie <ka...@plutoz.com>.
if you're referring to this :
> fullfill request on this host CassandraClient<crawler1:9160-2>
I am not adding anything. I am not even using port numbers. I thought
that was hector. I double checked all the config files I could think of
after I got your email again. I am not adding the '-2'.
On 08/29/2013 09:23 AM, Lewis John Mcgibbney wrote:
> no I don't think increasing the timeout will help as this is not the
> source of the problem. Cassandra has a very simple write path this is
> boosted by high write speeds and throughput. I would guess that your
> Cassandra pin e.g. http://host:9160-2 cannot be mapped to. is there any
> reason that you've added the trailing hyphen two?
> hath
> lewis
>
> On Wednesday, August 28, 2013, kaveh minooie <kaveh@plutoz.com
> <ma...@plutoz.com>> wrote:
> >
> > Hi every one,
> > this is an example of what I am getting :
> >
> > 2013-08-28 16:41:09,520 INFO
> org.apache.gora.mapreduce.GoraRecordWriter: Flushing the datastore after
> 10000 records
> > 2013-08-28 16:41:19,556 WARN
> me.prettyprint.cassandra.connection.HConnectionManager: Could not
> fullfill request on this host CassandraClient<crawler1:9160-2>
> > 2013-08-28 16:41:19,557 WARN
> me.prettyprint.cassandra.connection.HConnectionManager: Exception:
> > me.prettyprint.hector.api.exceptions.HTimedOutException:
> TimedOutException()
> > at
> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
--
Kaveh Minooie
Re: too many HTimedOutException in reduce tasks
Posted by Lewis John Mcgibbney <le...@gmail.com>.
no I don't think increasing the timeout will help as this is not the source
of the problem. Cassandra has a very simple write path this is boosted by
high write speeds and throughput. I would guess that your Cassandra pin
e.g. http://host:9160-2 cannot be mapped to. is there any reason that
you've added the trailing hyphen two?
hath
lewis
On Wednesday, August 28, 2013, kaveh minooie <ka...@plutoz.com> wrote:
>
> Hi every one,
> this is an example of what I am getting :
>
> 2013-08-28 16:41:09,520 INFO org.apache.gora.mapreduce.GoraRecordWriter:
Flushing the datastore after 10000 records
> 2013-08-28 16:41:19,556 WARN
me.prettyprint.cassandra.connection.HConnectionManager: Could not fullfill
request on this host CassandraClient<crawler1:9160-2>
> 2013-08-28 16:41:19,557 WARN
me.prettyprint.cassandra.connection.HConnectionManager: Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException:
TimedOutException()
> at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at
me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at
me.prettyprint.cassandra.model.MutatorImpl.insert(MutatorImpl.java:69)
> at
org.apache.gora.cassandra.store.HectorUtils.insertColumn(HectorUtils.java:52)
> at
org.apache.gora.cassandra.store.CassandraClient.addColumn(CassandraClient.java:197)
> at
org.apache.gora.cassandra.store.CassandraStore.addOrUpdateField(CassandraStore.java:353)
> at
org.apache.gora.cassandra.store.CassandraStore.flush(CassandraStore.java:228)
> at
org.apache.gora.mapreduce.GoraRecordWriter.write(GoraRecordWriter.java:70)
> at
org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:586)
> at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at
org.apache.nutch.crawl.DbUpdateReducer.reduce(DbUpdateReducer.java:202)
> at
org.apache.nutch.crawl.DbUpdateReducer.reduce(DbUpdateReducer.java:39)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
> at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: TimedOutException()
> at
org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at
org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at
org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at
me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at
me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 20 more
>
>
> this is happening in virtually all of my reduce tasks, and I think as a
result of this, I get this for about 15% of my reduce tasks:
>
> Task attempt_201308151359_0006_r_000014_0 failed to report status for
1800 seconds. Killing!
>
>
> is there anything that I can do to improve this situation? maybe
increasing the timeout time some where?
>
> thanks,
>
>
> --
> Kaveh Minooie
>
--
*Lewis*