map-reduce failure

[Or Yanay <or...@peer39.com>]

Hi all,
I am using 0.6.8 across 5 machines with ~30G of data on each machine.
I am trying to run a map-reduce query (Both with my own Java code and Pig) and failing after about 30 minutes (see stack trace and details below).
I have followed this wiki page<http://wiki.apache.org/cassandra/HadoopSupport> to increase ulimit to 32K which seemed to make things work better but I still fails.

Can anyone suggest anything that can help resolve this?
Thanks,
-Or


The Pig query is pretty simple:

grunt> rows = LOAD 'cassandra://Indexing/EdgeCache' USING CassandraStorage();
grunt> b = GROUP rows ALL;
grunt> c = FOREACH b GENERATE COUNT(rows.$0);
grunt> dump c;

the Java code is doing something very similar.

java.lang.RuntimeException: TimedOutException()
                at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:186)
                at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:236)
                at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:104)
                at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
                at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
                at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:98)
                at org.apache.cassandra.hadoop.pig.CassandraStorage.getNext(Unknown Source)
                at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:142)
                at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423)
                at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
                at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
                at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
                at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
Caused by: TimedOutException()
                at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:11094)
                at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:628)
                at org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:602)
                at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:164)
                ... 13 more

The Pig log:

Pig Stack Trace
---------------
ERROR 1066: Unable to open iterator for alias c

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias c
        at org.apache.pig.PigServer.openIterator(PigServer.java:521)
        at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:544)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:162)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:138)
        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:75)
        at org.apache.pig.Main.main(Main.java:357)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
        at org.apache.pig.PigServer.openIterator(PigServer.java:515)
        ... 6 more